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I.  INTRODUCTION 


A.  PURPOSE 

The  purpose  of  this  thesis  is  to  explain  the  benefits  and  rationale  of 
Software  Independent  Verification  and  Validation  (SIV&V)  to  Program 
Management  Offices  (PMO)  and  others.  Additionally,  this  thesis  serves  as  a 
tutorial,  providing  policy  and  guidance  documentation,  as  well  as,  suggested 
software  Computer-Aided  Software  Engineering  (CASE)  tools,  criteria,  and 
lessons  learned  to  implement  a  successful  SIV&V  program. 

B.  BACKGROUND 

The  costliest,  most  complex  and  critical  component  of  almost  every 
weapons  system  employed  by  the  Armed  Forces  of  the  United  States  is  the 
software.  The  software  is  that  key  component  of  a  weapon  system  that  allows  it 
to  perform  its  functions.  For  the  war-fighters  to  be  able  to  successfully  carry  out 
their  missions,  the  software  must  be  able  to  perform  in  its  operational 
environment(s)  per  requirements.  Failure  to  do  so  can  lead  to  mission  failure 
and  catastrophic  consequences  for  the  war-fighters  and  their  equipment.  An 
excellent  quote  that  illustrates  this  point  was  made  by  LTG  Robert  H.  Ludwig, 
and  simply  states  the  “Fly-by-Wire  F16C  ...  without  software,”  is  nothing  more 
than  "...  a  15-million  dollar  lawn  dart!”  (Department  of  Air  Force  1-14). 

When  the  National  Security  of  the  United  States  and  the  lives  of  its  men 
and  women  in  uniform  are  at  stake,  this  country,  especially  members  of  the 
Defense  community,  must  strive  to  provide  the  very  best  engineered  software 
products  that  money  and  a  disciplined  engineering  process  can  produce.  SIV&V 
becomes  a  critical  tool  in  verifying  and  validating  highly  complex  software  that 
helps  assure  the  war-fighters’  weapon  systems  will  be  operational  and  perform 
as  required  when  they  are  needed  most. 


1 


C.  RESEARCH  QUESTIONS 

1.  Primary  Research  Question 

What  are  the  benefits  of  and  rationale  for  PMOs  and  others  for  using 
SIV&V? 

2.  Secondary  Research  Questions 

What  software  CASE  tools  are  available  for  software  V&V? 

What  key  things  should  be  done  or  considered  when  conducting  SIV&V? 
What  are  the  SIV&V  process  steps? 

How  has  acquisition  reform  affected  SIV&V? 

What  lessons  have  been  learned  from  past  programs  that  have  utilized 
SIV&V? 

D.  SCOPE  OF  THESIS 

This  thesis  serves  as  a  tutorial  on  SIV&V  for  the  Program  Manager  and 
others  engaged  in  the  acquisition  of  systems  for  the  United  States  Armed  Forces. 
As  such  its  scope  is  limited  to  providing  a  brief  history  of  SIV&V,  highlighting 
some  policy  and  guidance  documentation,  discussing  acquisition  reform,  SIV&Vs 
importance  to  the  overall  success  of  producing  systems  that  are  both 
operationally  effective  and  suitable,  and  providing  an  understandable  practical 
methodology  for  conducting  SIV&V.  Several  real  world  examples  are  used  to 
convey  the  benefits  of  employing  SIV&V,  the  pitfalls  of  not  using  SIV&V,  and 
providing  lessons  learned  that  can  benefit  programs  that  apply  this  methodology. 

E.  METHODOLOGY 

1.  Data  Collection  Methodology 

The  data  collection  methodology  for  this  thesis  consisted  of  extensive 
research  of  available  SIV&V  materials  from  books,  the  internet  and  online  library 
sources,  government  and  other  policies,  regulations,  standards  and  handbooks. 
Additionally,  the  more  than  1 1  years  of  SIV&V  experience  of  one  of  the  authors, 
Mr.  Ashley  Mathis,  was  instrumental  in  bringing  this  information  together  in  an 
understandable,  concise  and  practical  form. 
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2.  Data  Analysis  Methodology 

All  of  the  data  was  analyzed  from  the  perspective  of  real  world  experience 
with  SIV&V.  The  data  analysis  was  conducted  to  determine  what  did  and  did  not 
work,  how  to  apply  or  not  to  apply  the  method,  why  to  apply  the  method,  and 
when  to  apply  the  method.  A  critical  eye  was  applied  to  looking  for  practical, 
straightforward  ways  in  which  to  apply  SIV&V  methods  and  lessons  learned  that 
could  be  easily  understood  and  applied  to  any  acquisition  program. 

3.  Conclusions  and  Recommendations  Methodology 

This  thesis  will  aid  PMOs  in  making  informed  decisions  when  faced  with 
political  and  budgetary  realities.  Thus,  the  conclusions  and  recommendations  of 
this  thesis  stemmed  from  a  careful  analysis  of  the  data  through  the  prism  of  real 
world  experiences.  The  objective  was  to  provide  a  clear  understanding  of  the 
need  for  and  the  benefits  of  a  properly  applied  SIV&V  methodology.  Armed  with 
this  information,  PMOs  will  have  understanding  of  a  powerful  tool  that  is  critical  to 
the  successful  acquisition  of  today’s  complex  software  centric  systems. 

F.  ORGANIZATION 

This  thesis  consists  of  four  chapters. 

Chapter  I  -  Introduction  -  This  chapter  establishes  the  purpose, 
background,  the  research  questions,  the  scope,  the  methodology,  organization, 
and  benefits  of  this  thesis. 

Chapter  II  -  SIV&V  Background  -  This  chapter  introduces  the  material, 
providing  a  description,  definitions,  and  a  history  of  SIV&V’s  evolution  in 
conjunction  with  the  technological  progress  of  computers  and  software.  Also 
addressed  are  some  guidelines  and  policies,  the  role,  and  the  future  challenges 
facing  SIV&V. 

Chapter  III  -  The  SIV&V  Guide  -  This  chapter  addresses  sizing  and  types 
of  SIV&V  agents,  metrics,  and  staffing  levels.  Additionally,  the  benefits  of  SIV&V 
are  addressed,  using  real  world  examples  to  illustrate  the  differences  between 
programs  that  used  SIV&V  and  ones  that  did  not.  Finally,  the  chapter  discusses 
how  to  apply  SIV&V  throughout  the  life  cycle  of  a  program,  the  CASE  tools  that 
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can  be  used  to  manage  the  effort,  the  impacts  of  acquisition  reform,  and  key 
strategies  for  conducting  SIV&V  successfully. 

Chapter  IV  -  Summary,  Recommendations,  and  Conclusions  -  This 
chapter  summarizes  the  answers  to  the  research  questions  from  chapter  I,  and 
recommends  an  area  for  further  study. 

Appendix  -  The  appendix  provides  a  chart  showing  the  Integrated 
Systems  Diagram  process  that  the  Software  Engineering  Directorate  uses.  This 
is  followed  by  the  List  of  References. 

G.  BENEFITS  OF  STUDY 

This  study  will  benefit  PMOs  and  others  who  are  acquiring  systems  in 
support  of  the  American  men  and  women  in  uniform.  The  PMOs  will  gain  an 
advantage  by  having  a  better  understanding  of  the  benefits  of  utilizing  an 
effectively  tailored  SIV&V  process  that  will  allow  them  to  produce  highly  reliable 
software  while  reducing  the  life  cycle  costs  of  their  programs.  Further,  the  PMOs 
will  have  a  better  understanding  of  the  process  steps,  methodologies,  and  tools 
that  are  critical  to  successfully  applying  SIV&V  to  their  programs.  Finally,  PMOs 
will  have  a  readily  available  reference  that  specifies  various  SIV&V  policies  and 
guidelines  and  where  they  can  be  found.  Ultimately,  it  is  the  warfighter  who  will 
reap  the  greatest  benefits  through  fielding  of  highly  reliable  systems  that  are 
operationally  effective  and  suitable. 
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II.  THE  SIV&V  BACKGROUND 


A.  INTRODUCTION 

“Modern  aerospace  and  defense  systems  incorporate  increasingly 
sophisticated  information  processing  and  control  systems”  (Department  of  Air 
Force  U-3).  Given  the  ever  increasing  complexity  of  these  systems,  policy 
makers  recognizing  the  necessity  for  and  the  benefits  of  SIV&V,  have  instituted 
policies,  regulations,  and  guidance  making  SIV&V  an  integral  part  of  modern 
systems  acquisition.  One  such  regulation  is  Department  of  Defense  (DoD) 
Directive  5000.1,  which  identifies  Software  Independent  Verification  and 
Validation  (SIV&V)  as  providing  the  Program  or  Project  Office  (PO)  with  an 
independent  assessment  of  the  software.  Thus,  it  becomes  the  responsibility  of 
the  SIV&V  agent  to  ensure  systems  operate  and  continue  to  demonstrate  a  high- 
level  of  reliability  throughout  their  life  cycles.  This  document  is  a  synopsis  of  the 
overarching  processes  and  justification  for  the  SIV&V  effort,  and  is  used  as  the 
top-level  guideline  for  all  SIV&V  activities.  The  SIV&V  Background  section  will 
discuss  the  nature  and  rationale  for  the  SIV&V  environment. 

B.  KEY  DEFINITIONS 

Independent:  The  use  of  a  team  that  is  separate  from  the  development 
and  development  management/oversight  teams  to  perform  V&V. 

Software  Verification:  “Are  we  building  the  right  thing  right?”  - 
Confirmation  by  examination  and  provision  of  objective  evidence  that  specified 
requirements  fulfill  the  output  of  a  particular  phase  of  development  and  meet  all 
the  input  requirements  for  that  phase. 

•  Verifies  software  architecture  design  based  upon  software 
requirements,  Software  Development  Plan  (SDP),  and  the  Software 
Design  Document  (SDD) 

•  Analyzes  interfaces,  data  flow,  exception  handling,  timing  budgets, 
memory  allocations 

•  Compares  code  to  design  and  development  requirements 
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Software  Validation:  “Are  we  building  the  right  thing?”  -  Establish 
objective  evidence  that  all  software  requirements  are  correctly  implemented, 
complete  and  are  traceable  to  system  requirements.  Software  validation  is  a 
design  verification  function  and  includes  all  of  the  verification  and  testing 
activities  conducted  throughout  the  software  life  cycle. 

•  Ensures  that  all  software  requirements  are  qualified  (certified)  through 
analysis,  inspection,  demonstration,  or  test 

•  SIV&V  assesses  software  capability  to  meet  specified  design  and 
performance  requirements 

Software  Independent  Verification  and  Validation  (SIV&V):  An 

independent  risk  reduction  processes  applied  to  operational  software  to  ensure 
the  prepared  code  is  properly  built  to  specifications  and  adequately  tested  for 
deployment  in  a  delivered  operational  system. 

C.  SIV&V  BACKGROUND 

1.  Historical 

To  truly  appreciate  and  comprehend  how  and  why  SIV&V  developed,  and 
the  critical  part  that  it  plays  in  the  development  of  today’s  complex  systems,  a 
brief  review  of  the  history  of  computers  and  software  development  is  needed. 

The  first  electronic  computer  was  developed  in  the  1940s  (Rombach  52). 
These  early  computers  did  not  separate  the  hardware  from  the  software,  as  they 
were  built  to  perform  one  task  or  solve  one  problem,  and  thus  were  switched  or 
hard-wired  to  perform  certain  tasks  (Robat  5-7).  By  the  1950s,  computers  had 
progressed  to  single-user  operating  systems,  supported  high-level  programming 
languages  such  as  COBOL  (Common  Business  Oriented  Language)  (Codasyl 
committee  1960),  and  FORTRAN  (FORmula  TRANslator)  (IBM  1952)  (Robat  11), 
and  multiple  computer  applications  (Rombach  52).  In  the  1960s  computers 
became  more  powerful,  supported  multi-user  operating  systems,  a  variety  of 
more  intricate  applications,  and  were  used  extensively  to  solve  ever  more 
complex  problems  (Rombach  52).  By  1968,  computers  and  software  had 
evolved  to  a  level  of  complexity  where  the  term  “The  Software  Crisis”  was 
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applied  for  the  first  time  at  a  NATO  conference  in  Garmisch  -Partenkirchen 
(Rombach  53),  to  describe  the  growing  problem  with  software  quality  and 
reliability.  The  ensuing  decades  of  the  70s,  80s,  and  90s  have  only  added  to  the 
problem.  As  computers  have  become  more  capable  and  the  software  more 
complex,  engineers  have  utilized  these  technical  advances  to  solve  ever  more 
sophisticated  and  intricate  problems.  As  a  result,  we  have  seen  software  grow 
from  a  relatively  small  contribution  to  system  development  cost  of  20%  in  the 
1950s,  to  80%  of  system  cost  in  the  1980s,  to  nearly  95%-100%  of  system  cost 
today  (Reiss  397-398).  This  change  in  cost  between  hardware  and  software  can 
be  explained  as  follows.  The  cost  of  the  hardware  for  the  large,  early  computers 
was  very  expensive;  typically,  very  few  were  built,  and  those  that  were  occupied 
the  space  of  several  rooms.  As  a  result  of  technological  advances  and  the 
utilization  of  mass  production  techniques,  computer  hardware  became  not  only 
more  capable  and  smaller,  but  also  much  cheaper.  Further,  in  the  past,  large 
software  programs  consisted  of  thousands  of  lines  of  code.  In  contrast,  today’s 
systems  consist  of  millions  of  lines  of  code.  Software  engineers  have  found  that 
as  lines  of  code  increased  arithmetically,  the  cost  and  complexity  of  the  software 
tended  to  increase  exponentially.  New  methodologies  and  techniques  were 
required  to  develop  and  manage  the  ever-increasing  size  of  these  software 
systems  (Reiss  397-398). 

From  its  infancy  in  the  1940s  to  today,  “...software  development  has 
evolved  from  small  tasks  involving  a  few  people  to  enormously  large  tasks 
involving  many  people”  (Tran  1-2).  Similarly  V&V  has  changed  from  an 
“...informal  process  performed  by  the  software  engineer  himself...”  to  a  “...highly 
formalized...”  “...separate  activity...”  conducted  by  “...organizations  independent 
of  the  software  developer...”  and  “...practiced  over  the  entire  software  life  cycle” 
(Tran  1-2). 

Given  the  historical  context  of  computing  and  software,  it  is  not  surprising 
that  V&V  and  IV&V  of  software  developed  at  a  much  later  date,  not  because  it 
was  planned,  but  rather  as  a  necessity  for  dealing  with  increasing  software 
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complexity  and  the  resulting  lack  of  quality  and  reliability.  In  the  earliest  days  of 
computers  and  software,  there  was  a  significant  “...lack  of  discipline  in  the 
software  development  process  ...”  (Persons  5-7).  This  early  “V&V”  process,  if 
you  could  call  it  such,  was  nothing  more  than  the  programmer  debugging  their 
code  (Shridhar  2-3).  As  software  began  to  mature  in  the  late  1950s,  the  U.S. 
Department  of  Defense  (DoD)  began  to  notice  that  as  their  systems  started 
incorporating  more  software,  three  issues  kept  making  repeat  appearances, 
namely,  budget  overruns,  schedule  delays,  and  technical  problems  (Food  For 
Thought  1-2).  These  recurring  issues  necessitated  that  a  more  formal  method  of 
developing  and  managing  software  be  established.  One  of  the  very  first  uses  of 
V&V  was  in  the  DoD  on  “...the  Atlas  Missile  Program  in  the  late  1950s”  (Food 
For  Thought  1-2). 

In  1962,  the  Air  Force  learned  an  expensive  lesson  with  the  loss  of  an 
Atlas  booster  and  its  Mariner  payload  because  of  a  simple  software  error 
(Shridhar  2-3).  This  failure  resulted  in  the  Air  Force  mandating  that  all  future 
mission  critical  software  would  require  independent  verification  (Shridhar  2-3).  It 
was  this  requirement  that  served  as  the  catalyst  for  what  we  know  today  as  IV&V 
(Shridhar  2-3). 

In  the  1970s,  as  software  began  to  migrate  to  the  commercial  sector, 
many  of  these  software  companies  began  to  experience  the  same  issues  that 
had  plagued  government  programs  a  decade  earlier  (Food  for  Thought  2-3).  It 
was  during  this  same  time  frame  that  "...  the  U.S.  Army  sponsored  the  first 
significant  such  IV&V  program  for  the  Safeguard  Anti-Ballistic  Missile  System. 
This  program  pushed  IV&V  from  a  fledgling  stage  to  being  a  mature  systems  and 
software  engineering  discipline....  It  was  from  this  effort  that  IV&V  became  well 
known  within  the  Department  of  Defense  and  aerospace  communities  as  an 
accepted  method  of  ensuring  better  quality,  performance,  and  reliability  of  critical 
systems....  By  the  mid-  to  late  1970s,  IV&V  was  rapidly  becoming  popular  and  in 
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some  cases  was  required  by  the  military  services,  especially  for  systems  that  has 
a  high  cost  of  failure  and  hence  were  able  to  justify  the  small  added  cost  of  IV&V” 
(Lewis  1992  xxiii). 

Even  as  the  Defense  and  Aerospace  communities  were  embracing  an 
evolving  SIV&V  as  a  viable  methodology  for  handling  software  quality  and 
reliability,  their  counterparts  in  the  commercial  sector  continued  to  experience 
quality,  budget,  schedule,  and  technical  problems  on  an  alarming  scale  as  late  as 
the  1980s  and  1990s  (Food  for  Thought  2-3).  Table  1  (Department  of  Air  Force 
2-7)  shows  some  examples  of  major  commercial  software  failures. 


Table  1 .  Major  Commercial  Software  Failures  (Department  of  Air  Force  2-7) 


YEAR 

PROJECT 

RESULTS 

1980s 

International  Telegraph  &  Telephone 
(ITT)  -  4  Switching  Systems 

40,000  Function  Point  System, 
$500M  Lost,  Cancelled 

1987 

California  Department  of  Motor 

Vehicles,  Automated  Vehicle/Drivers 
License  System 

3  (5,000  Function  Point  Size) 
Switches,  $30M  Lost, 

Cancelled 

1989 

State  of  Washington  -  Automated 

Social  Services  Caseworker  System 

7  Years  to  Build,  Failed  to 

Meet  User  Needs,  $20M  Lost, 
Cancelled 

1992 

American  Airlines  -  Flight  Booking 
System 

$165M  Lost,  Cancelled 

Over  the  past  60  years,  we  have  witnessed  the  birth,  growth,  and 
development  of  computers  and  software.  As  the  technology  matured,  and 
became  intertwined  throughout  our  modern  infrastructure,  its  very  complexity 
required  the  software  engineering  discipline  and  its  process  to  grow  and  mature 
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in  an  effort  to  control  and  manage  this  software.  A  natural  outgrowth  of  this 
evolution  was  the  need  for  SIV&V;  a  mindset,  processes,  and  a  set  of  tools 
developed  to  provide  software  engineers  and  PMs  the  capability  to  consistently 
produce  reliable,  quality  software. 

The  authors  expect  that  computers  and  software  technology  will  continue 
to  progress  and  mature,  becoming  ever  more  complex  and  sophisticated.  The 
future  successes  of  software  programs  will  become  ever  more  dependent  on  the 
SIV&V  process,  and  as  such,  will  continue  to  force  the  evolutionary  advancement 
of  SIV&V  so  that  it  remains  a  viable  and  effective  methodology  in  the  never- 
ending  battle  to  tame  and  manage  the  expanding  complex  nature  of  software. 

2.  SIV&V  Policy/Guidance 

Many  agencies  both  government  and  commercial  have  devised  policies, 
regulations,  and  standards  that  address  SIV&V.  Table  2,  listing  policies  and 
guidelines,  is  not  all-inclusive  but  serves  as  a  handy  reference  that  can  assist 
and  guide  the  reader  in  implementing  SIV&V  within  their  own  projects  and 
organizations. 


Table  2.  SIV&V  Policy  And  Guidance 


(Table  continues  on  following  pages) 


Policy/Regulation/Standard/ 

Other 

Agency 

Website/Comments 

AFSC/AFLCP  800-5  “Software 
Independent  Verification  and 
Validation” 

US  Department  of 
the  Air  Force  (AF) 

http://seqoldmine.ppi- 
int.com/menu  quides.htm 

AFSMC  Regulation  800-26 
“Independent  Verification  and 
Validation” 

US  Department  of 
the  Air  Force, 
Space  and  Missile 
Systems  Center 
(AFMC) 

http://www.fas.orq/spp/militarv/docop 

s/s  mc/i  vv26.htm 

AFI  16-1001  “Verification, 
Validation,  and  Accreditation” 

US  Department  of 
the  Air  Force 

http://www.e- 

publishinq.af.mil/pubfiles/af/16/afi16- 

1 001/afil  6-1 001  .pdf 

ESD-TR-326  “Software 
Acquisition  Management 
Guidebook:  Validation  and 
Certification” 

US  Department  of 
the  Air  Force, 
Electronic  Systems 
Division, 
Hanscomb  AFB 

http://stinet.dtic.  mil/oai/oai?&verb=qet 

Record&metadataPrefix=html&identifi 

er=ADA053039 
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IEEE  1012-2004  “IEEE  Standard 
for  Software  Verification  and 
Validation” 

Institute  for 
Electrical  and 
Electronics 
Engineers 

http://www.ieee.orci/web/standards/ho 

me/index. html 

ANSI/IEEE  1074-2006  “IEEE 
Standard  for  Developing  a 
Software  Project  Life  Cycle 
Process” 

American  National 
Standards 
Institute/Institute  for 
Electrical  and 
Electronics 
Engineers 

http://www.ieee.orq/web/standards/ho 

me/index. html 

http://webstore.ansi.orq/ansidocstore/ 

product. asp?sku=1 074%2D1 997 

IEEE  1059-1993  “IEEE  Guide  for 
Software  Verification  and 
Validation  Plans” 

Institute  for 
Electrical  and 
Electronics 
Engineers 

http://www.ieee.orq/web/standards/ho 

me/index. html 

NPD  8730.4  NASA  Policy 
Directive 

National 

Aeronautics  and 
Space 

Administration 

www.ivv.nasa.qov/foremplovees/polic 

vplans.php 

Established  NASA  Policy  for 
Independent  Verification  and 

Validation.  Replaced  by  2820. 1C. 

NPD  2820. 1C  NASA  Policy 
Directive 

National 

Aeronautics  and 
Space 

Administration 

www.ivv.nasa.qov/foremplovees/polic 

vplans.php 

NASA  Independent  Verification  and 
Validation  Policy 

“Independent  Verification  and 
Validation  Implementation  Plan 
2003-2008” 

National 

Aeronautics  and 
Space 

Administration 

www.ivv.nasa.qov/foremplovees/polic 

vplans.php 

“Independent  Verification  and 
Validation  Implementation  Plan 
2005-2010” 

National 

Aeronautics  and 
Space 

Administration 

www.ivv.nasa.qov/foremplovees/polic 

vplans.php 

NASA  Ol G  IG-03-01 1 
“Independent  Verification  and 
Validation  of  Software” 

National 

Aeronautics  and 
Space 

Administration 

http://oiq.nasa.qov/audits/reports/FYO 

3/odfs/io-03-011.odf 

NASA  Office  of  Inspector  General 
Audit  Report 

NASA-STD-8739.8  “Software 
Assurance  Standard” 

National 

Aeronautics  and 
Space 

Administration 

http://www.hq.nasa.qov/office/codeq/ 

doctree/87398.pdf 

NASA-STD-871 9. 1 3A  “Software 
Safety” 

National 

Aeronautics  and 
Space 

Administration 

http://satc.qsfc.nasa.qov/assure/nss8 

719  13.html 

“Program  Plan  for  the  NASA 
Software  Independent 
Verification  and  Validation 
Program”  Rev  1  May  04 

National 

Aeronautics  and 
Space 

Administration 

www.ivv.nasa.qov/foremplovees/polic 

vplans.php 
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NASA-GB-002-95  “Formal 
Methods  Specification  and 
Verification  Guidebook  for 
Software  and  Computer 
Systems” 

National 

Aeronautics  and 
Space 

Administration 

http://www.finq.edu.uv/inco/cirupos/mf 

/TPPSF/Biblioqrafia/fmquidel.pdf 

ANSI/ANS  10.4-1987;R1998 
“Guidelines  for  the  Verification 
and  Validation  of  Scientific  and 
Engineering  Computer  Programs 
for  the  Nuclear  Industry” 

American  National 
Standards  Institute/ 
American  Nuclear 
Society 

http://www.ans.orq/store/vi-240150 

BSR/AAMI  SW76- 
200x  “Software  Verification  and 
Validation  for  High-risk  Medical 
Devices” 

Association  for  the 
Advancement  of 
Medical 

Instrumentation 

http://www.nssn.orq/search/DetailRes 

ults.aspx?docid=41 766&selnode 

FHWA  Handbook  VI  .2 
“Verification,  Validation,  and 
Evaluation  of  Expert  Systems: 

An  FHWA  Handbook” 

Federal  Highway 
Administration 

h  tt  p  ://www .  tf  h  rc .  q  o  v/a  d  va  n  c/vve/cove 

r.htm 

h  tt  p  ://www .  tf  h  rc .  q  o  v/a  d  va  n  c/vve/toc .  h 

tm 

FIPSPUB  101  “Guideline  for  Life 
Cycle  Validation,  Verification, 
and  Testing  of  Computer 
Software” 

US  Department  of 
Commerce, 
National  Bureau  of 
Standards 

Federal  Information  Processing 

Standards  (FIPS) 

http://www.itl.nist.qov/fipspubs/withdr 

aw.htm 

Withdrawn  and  replaced  by  industry 
standards.  Can  still  be  bought  from 
http://www.nssn.orq/search/DetailRes 

ults.asox?docid=263282&selnode 

FIPSPUB  132  “Guideline  for 
Software  Verification  and 
Validation  Plans” 

US  Department  of 
Commerce, 
National  Bureau  of 
Standards 

Federal  Information  Processing 

Standards  (FIPS) 

http://www.itl.nist.qov/fipspubs/withdr 

aw.htm 

Withdrawn  and  replaced  by  industry 
standards  (IEEE  1012). 

NBS  Special  Publication  500-93 
“Software  Validation, 
Verification,  and  Testing 
Technique  and  Tool  Reference 
Guide” 

US  Department  of 
Commerce, 
National  Bureau  of 
Standards 

http://librarv.nist.qov/uhtbin/cqisirsiA/ 

weuQqvAsh/NIST/1 17380059/123 

NIST  Special  Publication  500- 
234  “Reference  Information  for 
the  Software  Verification  and 
Validation  Process” 

US  Department  of 
Commerce, 
National  Institute  of 
Standards  and 
Technology 

http://hissa.nist.qov/HHRFdata/Artifac 

ts/ITLdoc/234/val-proc.html 

NIST  Special  Publication  500- 
165  “Software  Verification  and 
Validation:  Its  Role  in  Computer 
Assurance  and  Its  Relationship 
with  Software  Project 
Management  Standards” 

US  Department  of 
Commerce, 
National  Institute  of 
Standards  and 
Technology 

http://librarv.nist. qov/uhtbin/cqisirsi/V 

weuQqvAsh/NIST/1 17380059/123 
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NIST  Special  Publication  500- 
223  “A  Framework  for  the 
Development  and  Assurance  of 
High  Integrity  Software” 

US  Department  of 
Commerce, 
National  Institute  of 
Standards  and 
Technology 

http://hissa.nist.ciov/publications/sp22 

3 / 

NUREG/CR-631 6  Volumes  1-8 
“Guidelines  for  the  Verification 
and  Validation  of  Expert  System 
Software  and  Conventional 
Software” 

US  Nuclear 
Regulatory 
Commission 

http://www.osti.aov/enerqvcitations/se 

archresults. isp?Author=Mirskv,+S.M. 

This  report  is  an  excellent  source  of 
information. 

PAM  5-11  Verification, 
Validation,  and  Accreditation  of 
Army  Models  and  Simulations” 

US  Department  of 
the  Army 

http://www.armv.mil/usapa/epubs/pdf/ 

o5  11. odf 

SEI-CM-13-1.1  “Introduction  to 
Software  Verification  and 
Validation  Module” 

Carnegie  Mellon 
University, 
Software 
Engineering 
Institute 

http://www.sei.cmu.edU/publications/d 

ocuments/cms/cm.01 3.html 

NASA  Briefing  “Software 
Independent  Verification  and 
Validation” 

National 

Aeronautics  and 
Space 

Administration 

A  briefing  on  how  to  conduct  SIV&V. 
http://ses.asfc.nasa.qov/ses  data  20 

01/010307  Bruner  IVV.oot 

NASA  SLP  IVV  09-1  Rev.  1 
Effective  March  2006 

National 

Aeronautics  and 
Space 

Administration 

This  is  a  System  Level  Procedure 
(SLP). 

http://ims.ivv.nasa.qov/sharedfiles/do 

cuments/IVV  09-1.doc 

NASA  PD-ED-1228 

“Independent  Verification  and 
Validation  of  Embedded 

Software” 

National 

Aeronautics  and 
Space 

Administration 

NASA  preferred  reliability  practice. 
http://klabs.orq/DEI/References/desiq 

n  quidelines/desiqn  series/1 228msfc 

.pdf 

3.  The  Role  of  SIV&V 

The  primary  role  of  SIV&V  is  to  provide  the  Program  Director  (PD)  or 
Management  with  an  independent  assessment  capability  to  guarantee: 

•  Software  is  developed  and  tested  to  ensure  high  confidence  in  the 
system  and  component  capabilities 

•  Software  is  mature  and  dependable 

•  Technical  issues  and  trends  are  identified  in  a  timely  manner  for 
Management  focus 

•  Risk  reduction  of  Program,  Element  or  Component  failures  due  to 

operational  or  simulation  software  defect(s) 
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These  activities  include  software  design  evaluation,  code  inspections, 
code  assessments,  and  independent  test  reviews.  The  overall  objective  of 
software  V&V  is  to  insure  that  the  product  is  free  from  failures  and  meets  its 
user’s  requirements  and  expectations. 

Broadly  speaking,  it  can  be  stated  that  some  level  of  SIV&V  should  be 
used  on  the  following  classes  of  systems  (Defense  Acquisition  University  386): 

•  Real-time  critical  software  systems  that  must  work  every  time  they  are 
used 

•  Programs  having  critical  outputs  that  cannot  be  verified  on  every  run 

•  Programs  having  a  high  cost  of  failure  in  terms  of  human  life,  national 
security  or  money 

•  Software  for  which  the  cost  of  error  detection  through  operational  use 
or  testing  exceeds  the  cost  of  employing  SIV&V 

•  Software  for  which  the  cost  of  support  is  expected  to  exceed  the  cost 
of  using  SIV&V 

Thus,  the  description  of  SIV&V  as  “cheap  insurance”  becomes  patently 
obvious,  particularly  when  one  considers  the  consequences  of  failure  of  critical 
software,  especially  when  lives  are  at  stake. 

4.  SIV&V  Challenges 

In  the  previous  three  sections  we  have  discussed  the  emerging  need  of 
SIV&V  from  a  historical  perspective,  the  policy  and  guidelines  developed  by  both 
the  government  and  commercial  entities,  and  the  primary  role  and  uses  of 
SIV&V.  Given  that  the  reader  accepts  the  premise  of  the  last  three  sections,  and 
understands  the  need  for  and  uses  of  SIV&V,  the  challenge  lies  in  how  to 
effectively  apply  a  SIV&V  strategy  to  existing  acquisitions  as  well  as 
incorporating  SIV&V  as  an  integral  part  of  new  acquisitions. 

The  first  part  of  this  challenge  is  to  educate  management  on  the  what, 
how,  and  necessity  of  SIV&V,  to  get  the  buy  in  required  for  implementation. 
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The  second  part  of  this  challenge  is  to  find  the  financial  resources  required 
to  carry  out  SIV&V  and  to  allocate  them  appropriately. 

The  third  part  of  this  challenge  is  to  educate  the  workforce  in  the  SIV&V 
process;  that  is  to  understand  when  it  makes  sense,  where  it  makes  sense,  how 
to  tailor  it,  how  to  implement  it,  and  how  to  utilize  it  to  improve  their  software 
processes  and  products. 

The  fourth  part  of  this  challenge  is  to  understand  that  SIV&V  is  not  the 
proverbial  “silver  bullet”  that  will  fix  poor  software  development  processes,  poor 
requirements  development  and  control,  poor  configuration  management,  poor 
systems/software  engineering,  poor  quality  control,  or  poor  documentation.  It  is 
however,  a  method  of  “cheap  insurance,”  that  will  allow  a  good  software  process 
to  be  even  better  by  finding  problems  before  software  gets  deployed  to  the  user. 

The  final  part  of  this  challenge  lies  in  understanding  the  nature  of  the 
SIV&V  process  as  one  that  is  constantly  evolving  and  developing  in  an  effort  to 
keep  pace  with  the  rapidly  changing  world  of  software  development.  This  will 
necessitate  that  those  utilizing  SIV&V  and  SIV&V  agents  stay  current  with 
developments  in  the  software  and  SIV&V  fields  and  continually  educate  their 
workforces  so  that  they  remain  both  effective  and  relevant. 
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III.  THE  SIV&V  GUIDE 


A.  SIV&V  SIZING  AND  TYPES 

1.  Introduction 

Software  IV&V  is  indeed  an  important  aspect  of  developing  quality 
software.  In  many  programs,  scarce  amounts  of  funds  are  allocated  to  SIV&V 
efforts.  Thus,  each  endeavor  should  be  tailored  so  that  it  corresponds  with  the 
level  of  criticality  of  the  software  development  effort.  This  section  will  discuss 
and  describe  the  different  types  and  sizing  metrics  of  SIV&V  agents 

2.  Survey  of  SIV&V  Metrics 

Several  key  SIV&V  metrics  were  established  for  comparing  the  size, 
responsibility,  cost,  and  performance  of  both  the  respective  SIV&V  teams  and  the 
Developer  across  various  systems.  The  cumulative  data  is  presented  in  Table  3. 
Names  of  actual  programs  and  associated  SIV&V  agents  were  changed  to 
protect  the  confidential  nature  of  the  information. 
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Table  3.  SIV&V  Metrics 


SIV&V  Metric 

System  A 

System  B 

System  C 

System  D 

Percent  of  SIV&V  Budget  to 

Development  Budget 

6.6% 

4.9% 

4% 

8% 

Percent  of  SIV&V  LOE  to 

Development  LOE 

4.3% 

8.2% 

5.7% 

15% 

Ratio  of  SIV&V  LOE  to  #  of  KSLOC 

1:72 

1:44 

N/A 

1:13 

Ratio  of  SIV&V  LOE  to  #  of  SW 

Requirements 

1:355 

1:525 

N/A 

1:350 

Development  Cost  per  SLOC 

$150 

$275 

N/A 

$125 

Average  Cost  to  Fix  Error 

(Before  SW  Release) 

Est.  $0.5K 

N/A 

N/A 

$2.5  per 

SLOC 

Average  Cost  to  Fix  Defect 

(After  SW  Release) 

Est. 

$1 K 

Est. 

$20. 6K 

N/A 

$12.5  per 

SLOC 

Ratio  of  SIV&V  LOE  Cost  to 

Developer  LOE  Cost 

1:1.54 

1:1.7 

N/A 

1:1.89 

(N/A  -  Not  Available) 


The  metrics  indicate  several  trends  common  across  all  of  the  surveyed 
efforts.  Items  of  interest  include: 

•  All  of  the  SIV&V  efforts  are  budgeted  less  than  10%  of  the  software 
development  budget. 

•  The  majority  of  the  SIV&V  efforts  have  staffing  levels  less  than  10%  of 
that  of  the  Developer. 

•  On  average,  each  SIV&V  analyst  is  responsible  for  410  software 
requirements  and/or  43  KSLOC. 

•  Average  SIV&V  labor  rates  are  approximately  59%  of  the  Developer 
rates. 
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As  indicated  above  most  SIV&V  groups  are  not  properly  resourced  and 
operate  in  a  degraded  mode.  However,  even  in  a  degraded  mode  of  operation  or 
service  they  always  provide  benefits  to  the  program  and  development  effort. 
From  the  research  provided  above,  each  individual  on  the  SIV&V  team  is 
responsible  for  approximately  300  -  550  requirements  and  10-98  thousand 
SLOC.  These  metrics  can  be  used  for  future  cost  and  resource  estimation 
purposes. 

3.  SIV&V  Staffing  Levels 

There  are  four  generally  accepted  types  (or  levels)  of  SIV&V  as 
documented  by  Lewis  (Lewis  1992  12-13): 

•  Full,  In-Phase  SIV&V  -  A  comprehensive  program  spanning 
requirements  phase  through  post-deployment  support.  SIV&V  costs 
for  this  type  are  typically  8-17%  of  the  total  program  software 
development  budget.  The  System  D  SIV&V  program  at  Software 
Engineering  Directorate  (SED)  is  an  example  of  the  complexity  of  a 
Full,  In-Phase  SIV&V. 

•  Partial  SIV&V  -  A  less  comprehensive  program  than  Full  SIV&V, 
Partial  SIV&V  begins  during  the  design  or  early  coding  phases  and  has 
limited  involvement  with  requirements  analysis.  SIV&V  costs  are 
typically  6-13%  of  the  total  software  development  budget. 

•  Endgame  SIV&V  -  This  level  of  SIV&V  is  focused  primarily  on  the  test 
and  integration  phase.  SIV&V  costs  are  typically  2-7%  of  the  total 
software  development  budget.  The  GMD  programs  at  SED  (Sea 
Based  X-band  Radar,  Embedded  Test,  and  Ground  Based  Interceptor) 
are  examples  of  Endgame  SIV&V  support. 

•  Audit-level  SIV&V  -  This  is  often  referred  to  as  “over-the-shoulder” 
SIV&V  and  is  a  minimal  effort.  Often,  a  tiger  team  is  used  to  determine 
the  adequacy  of  the  software  development  process. 

Additionally,  Lewis  asserts,  "Most  past  SIV&V  programs  have  cost 
between  2%  and  18%  of  the  software  development  cost.  The  higher-cost 
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programs  invariably  had  hardware  and/or  software  development  costs  like 
simulations  and  tools  embedded  in  them.  For  beginning  SIV&V  cost  estimates, 
try  to  begin  between  8%  and  10%  of  the  software  development  cost  estimate. 
SIV&V  programs  that  are  funded  at  less  than  4%  to  5%  of  the  development  cost 
will  have  to  begin  to  delete  some  routinely  performed  tasks.”  (Lewis  1992  280) 

During  the  course  of  this  study,  it  was  found  that  SIV&V  levels  were 
typically  that  of  the  Endgame  variety,  and  the  costs  for  the  SIV&V  programs 
varied  from  2%  to  10%  of  the  weapon  system’s  overall  software  budget,  with  an 
average  of  5.3%.  After  analyzing  the  levels  of  SIV&V  and  the  activities 
conducted  on  these  programs,  and  given  the  relatively  low  percentage  of 
program  funding  provided  for  SIV&V,  most  of  the  programs  had  to  delete  some 
routine  tasks  in  order  to  provide  the  most  positive  impact  to  the  program  with 
limited  funding. 

For  a  secondary  reference,  Figure  1  below  provides  a  general 
recommendation  for  the  percentage  of  technical  support  in  relation  to  the  size  of 
the  program  as  depicted  by  the  AMC  School  of  Engineering  and  Logistics.  Most 
SIV&V  programs  are  usually  at  the  lower  end  of  this  range  regardless  of  the 
program  size. 
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Figure  1 .  Technical  Support  And  Program  Size  (Missile  Defense  Agency  15) 

B.  THE  BENEFITS  OF  SIV&V 

1.  Introduction 

Without  question,  the  majority  of  weapon  systems  have  become 
increasingly  software  centric  with  each  generation  of  system  that  is  fielded.  The 
allocation  of  budget  to  software  engineering  has  proportionally  grown  as  the 
systems  have  expanded  and  matured.  A  key  program  management  technique 
employed  by  many  project  offices  that  has  successfully  reduced  software  risks 
and  increased  confidence  in  performance  attainment  is  Software  Independent 
Verification  and  Validation  (SIV&V). 

A  significant  portion  of  the  activities  at  the  U.S.  Army,  Research, 
Development,  and  Engineering  Command  (RDECOM),  Aviation  and  Missile 
Research,  Development  and  Engineering  Center  (AMRDEC),  Software 
Engineering  Directorate  (SED)  at  Redstone  Arsenal,  Alabama,  are  dedicated  to 
the  execution  of  SIV&V  programs  for  a  cadre  of  important  weapon  systems. 
These  programs  vary  in  budget,  size,  and  complexity.  The  weapon  systems 
represent  the  full  range  of  phases  in  the  development  cycle  and  fielding. 
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The  purpose  of  Section  B  is  the  following:  1)  Provide  data  that  supports 
the  value  of  implementing  a  SIV&V  program  on  a  software  intensive  system;  2) 
Provide  recommendations  based  on  SIV&V  success  stories  and  lessons  learned 
to  help  improve  the  acquisition  of  software  intensive  systems;  3)  Recommend  an 
approach  for  increasing  the  probability  of  successful  software  program 
development. 

2.  Costs  Associated  with  Discovered  Errors 

A  fundamental  tenant  of  SIV&V  is  that  the  earlier  in  the  software  life  cycle 
in  which  an  error  is  detected,  the  cheaper  it  is  to  fix.  This  is  usually  expressed  in 
terms  of  the  software  phase  in  which  the  error  is  detected  versus  the  cost  to  fix 
the  error.  The  same  relationship  can  be  expressed  in  relative  terms  such  as,  a 
requirements  error  discovered  during  operation  costs  approximately  200  times 
more  to  fix  than  finding  the  error  during  requirements  definition  (Boehm  1981). 
SIV&V  studies  use  this  approach  to  quantify  SIV&V  savings.  The  most 
universally  utilized  relationship  is  from  Barry  Boehm’s  classic  textbook,  Software 
Engineering  Economics  (Boehm  1981 ).  His  study  showed  that  the  relationship  is 
logarithmic.  This  is  shown  in  the  Figure  2: 
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Figure  2.  Cost  To  Fix  Defect  Versus  Life  Cycle  Phase  (Boehm  1981  40) 


Since  the  purpose  of  this  section  is  to  calculate  such  costs,  a 
mathematical  representation  of  the  figure  may  be  useful.  By  using  the  midpoint 
of  the  six  phases  projected  to  the  given  line  and  numbered  1  to  6,  a  table  of 
values  can  be  generated  and  an  exponential  equation  calculated.  The 
approximate  equation  is: 

y  =  .717e921x 

A  table  lookup  approach  will  also  work.  This  is  presented  in  the  Table  6, 
“Study  Multipliers,”  later  in  this  document. 

Experience  shows  that  not  all  errors  are  created  equal.  This  is  why  all 
Software  Problem  Report  (SPR)  systems  utilize  a  priority  scheme.  The  most 
commonly  used  scale  is  1  to  5,  where  1  indicates  a  safety-critical  item  or 
prevents  mission  accomplishment,  and  5,  which  typically  indicates  an 
insignificant  documentation  issue. 
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How  does  this  relate  to  the  value  of  SIV&V?  Simply  stated,  an  SIV&V 
team  that  routinely  finds  higher  priority  defects  is  more  valuable  than  a  team  that 
routinely  finds  insignificant  documentation  errors.  A  simplistic  approach  to 
handle  this  would  be  to  include  only  Priority  1,  2,  and  3  SPRs  in  the  value-added 
calculation.  Although  this  approach  simplifies  the  calculation,  it  ignores  two  key 
points.  First  ,a  single  Priority  1  SPR  might  be  much  more  valuable  than  several 
Priority  3  SPRs  combined.  Second,  finding  many  lower  level  SPRs  is  still  of 
value.  Again,  not  all  defects  and  SPRs  are  created  equal.  In  addition,  not  all 
SIV&V  findings  are  associated  exclusively  with  an  SPR.  Therefore,  basing  the 
calculation  solely  on  Priority  1, 2,  and  3  SPRs  misses  some  value-added  features 
of  an  SIV&V  team. 

A  more  realistic  approach  to  determine  SIV&V  value  is  to  draw  upon  a 
logarithmic  weighting  calculation  similar  to  the  relative  cost  calculation  approach 
indicated  previously.  A  simple  three  “bin”  approach  for  mission  critical  findings, 
major  findings,  and  findings  scaled  at  100  units,  10  units,  and  1  unit,  respectively, 
gives  adequate  granularity  with  a  simple  calculation.  It  is  then  straightforward  to 
calculate  total  units  divided  by  level-of-effort  (LOE)  to  obtain  value  per  person 
during  a  period  of  time.  Summary  metrics  for  multiple  groups  and  trend  graphs 
can  then  be  easily  generated. 

Please  note  the  importance  of  the  LOE  being  included.  For  example,  a 
team  of  five  people  finding  ten  problems  is  more  valuable  than  a  team  of  fifty 
people  finding  the  same  ten  problems.  Calculating  the  value  over  a  constant 
period  is  necessary  to  show  trends  and  process  improvement.  The  use  of  the 
generic  term  “unit”  is  also  intentional.  By  eliminating  discussion  of  costs  up  front, 
the  true  issue  of  productivity  (units)  is  highlighted,  and  is  not  clouded  by  dollars. 

The  following  example  from  System  D  at  SED  illustrates  this  method.  A 
team  of  30  finds  3  mission  critical  findings,  51  major  findings,  and  14  basic 
findings.  The  calculation  of  value-added  is  simply: 


(3*100  +  51*10  +  14*1)  /  30  =  824  units/ 30  LOE  =  27.5  units  per  person 
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Assigning  findings  into  three  “bins”  is  similar  to  assigning  priorities  to 
SPRs.  Mission  critical  findings  would  include  such  findings  as:  mission  saving, 
safety  critical,  system-of-systems  improvement,  and  DoD/Army-level  impact. 
Major  findings  could  include:  test  failures  prevented/detected,  major  process 
problems,  significant  cost  savings,  production  improvement,  and  significant 
interoperability  issues. 

The  list  for  findings  is  more  general  and  flexible  to  allow  the  full  scope  of 
SIV&V  value-added  to  be  included.  This  is  allowed  in  this  approach  since  each 
finding  is  only  worth  one  unit. 

SPRs  identified  by  the  SIV&V  team  would  be  included  as  follows: 

•  Category  1  SPR  =  Mission  Critical  Findings 

•  Category  2  SPR  =  Major  Findings 

•  Category  3  SPR  =  Findings 

In  summary,  the  appeal  of  this  approach  is  multifaceted.  It  is  simple  to 
use.  It  can  be  used  on  small  or  large  programs.  It  can  be  used  on  both  simple 
and  complex  programs.  And  most  importantly,  it  can  be  used  to  relate  the  value- 
added  of  SIV&V  between  dissimilar  programs. 

3.  Return  On  Investment  (ROI) 

In  order  to  show  ROI  for  SIV&V  costs,  the  “I”  in  ROI  must  be  quantified. 
SIV&V  costs  are  dependent  upon  software  program  complexity,  size,  and  needs, 
as  well  as,  the  breadth  and  depth  of  involvement  required  of  the  SIV&V  team 
performing  the  service.  As  a  general  rule,  total  SIV&V  costs  are  dependent  upon 
the  extent  of  SIV&V  performed,  and  in  which  phase  of  the  program  SIV&V  is 
begun. 

An  accepted  management  metric  that  is  used  to  determine  the  worth  of  an 
investment  is  ROI.  It  is  also  known  as  the  benefit-to-cost  ratio.  Even  though 
there  are  those  who  believe  that  only  a  small  impact  is  attained  by  SIV&V,  there 
is  sufficient  evidence  that  numerous  DoD  and  National  Aeronautics  and  Space 
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Administration  (NASA)  software  programs  have  implemented  successful  SIV&V 
programs  that  have  delivered  positive  ROIs. 

This  study  selected  two  programs,  Systems  B  and  D,  from  Table  3  “SIV&V 
Metrics”  in  Section  A  2,  in  an  effort  to  survey  the  ROI  for  the  weapon  systems 
employing  SIV&V.  Although  many  ROI  calculations  exist,  this  derived  calculation 
would  best  fit  our  software  situation. 

The  derived  formula, 

ROI  =  (A-(B-  C))  /  Overall  SIV&V  Costs 

Where:  A  =  Costs  avoided  by  intercepting  defect 

B  =  Costs  to  process  the  defect 
C  =  Costs  to  correct  the  defect 
is  used  to  calculate  the  SIV&V  ROI. 


The  general  approach  for  calculating  ROI  was  to  first  calculate  the 
average  cost  per  delivered  source  line  of  code  (SLOC).  Second,  the 
development  phase  in  which  SIV&V  detected  the  error  was  determined.  For  this 
study,  all  SIV&V  detected  errors  found  occurred  in  the  “Test  Phase.”  It  goes 
without  saying,  that  waiting  until  testing  to  identify  and  remove  problems  negates 
most  of  the  benefits  associated  with  SIV&V.  Third,  a  determination  was  made  to 
find  in  which  phase  the  error  originated.  Using  multipliers  based  on  documented, 
historical  data,  the  cost  to  process  and  correct  the  defect  was  calculated.  Two 
different  multipliers  were  used  in  order  to  show  that  there  is  disagreement 
between  the  relative  costs  to  correct  defects  by  phase.  Finally,  the  cost 
avoidance  was  determined  assuming  that  the  error  would  not  have  been  caught 
until  deployment  if  no  SIV&V  had  been  utilized.  The  inputs  to  the  ROI  calculation 
are  addressed  below. 
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a.  Average  Source  Line  of  Code  (SLOC)  Development  Cost 

First,  a  basis  for  cost  to  repair  a  defect  was  needed.  It  was  decided 
that  the  developer’s  cost  per  delivered  SLOC  would  be  used  as  an  input  for 
determining  cost  avoidance  and  defect  correction.  This  relationship  can  be 
described  as  follows: 

$/  SLOC  =  Total  Development  Software  Budget  /  Delivered  SLOC 


Our  study  found  that  delivered  SLOC  costs  vary  based  upon  a 
variety  of  factors  such  as  programming  language,  experience  of  the  Developer, 
complexity  of  the  system,  and  location  of  the  Developer’s  organization.  We 
found  that  the  development  cost  for  the  systems  analyzed  was: 

System  B  =  $275  per  delivered  SLOC 
System  D  =  $125  per  delivered  SLOC 


The  important  thing  to  note  is  that  the  delivered  code  costs  can 
become  much  higher  if  SIV&V  is  not  employed  early  in  the  lifecycle.  An  absolute 
dollar  cost  is  not  always  the  most  important  central  aspect,  but  rather  the  costs  of 
changes  and  errors  to  total  cost  which  are  phase  dependent.  Details  of  this 
research  are  demonstrated  in  the  paragraphs  below. 

b.  Analysis  of  SIV&V  Software  Trouble  Reports 
Corrected  and  Implemented  by  the  Developer 

For  the  systems  analyzed  for  this  study,  Software  Trouble  Reports 
(STRs)  or  Software  Change  Requests  (SCRs)  were  accepted,  analyzed, 
corrected,  and  implemented  by  the  developer.  The  STRs  were  assessed  as  to 
how  the  developer  classified  each  anomaly  as  to  the  phase  in  which  the  error 
originated.  The  classification  of  phase  originated  is  shown  in  Tables  4  and  5 
below: 
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Table  4.  System  B 


Phase  Originated 

#  STRs 

SLOC 

Requirements 

9 

24 

Design 

2 

38 

Code  and  Unit  Test 

48 

567 

TOTALS 

59 

629 

Table  5.  System  D 


Phase  Originated 

#  STRs 

SLOC 

Requirements 

74 

4154 

Design,  Code,  Unit  Test 

261 

1440 

Sys  Test  &  Integration 

12 

49 

TOTALS 

347 

5643 

From  the  metrics  in  the  above  tables,  one  can  identify  the  large 
difference  in  the  number  of  STRs  submitted  by  the  different  systems.  System  B 
was  only  4.9%  of  the  developer  budget  and  System  D  was  8.0%.  This  data 
indicates  that  more  problems  are  typically  found  when  a  larger  SIV&V  agent  is 
established.  Likewise,  the  data  reveals  where  increased  emphasis  must  be 
placed,  namely  Design,  Code,  and  Unit  Test  phases.  Based  on  our  examples  we 
believe  that  most  programs  would  experience  similar  results. 

c.  Multipliers  for  Relative  Cost  to  Correct  Defects 
Two  different  multiplier  sources  were  utilized  for  this  study.  In 
Table  6  below,  the  first  one  shown  was  developed  by  Barry  Boehm  based  on  an 
old  study  of  his  which  analyzed  three  different  systems  (Boehm  1989  206).  The 
other  source  was  from  the  Titan  study  (Barber  3). 
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Table  6.  Study  Multipliers 


Phase  Detected 

Phase  Originated 

Boehm 

Titan 

Multiplier 

Multiplier 

Requirements 

Requirements 

1 

1 

Design 

Requirements 

3 

5 

Design 

Design 

2 

1 

Code  &  Unit  Test 

Requirements 

9 

10 

Code  &  Unit  Test 

Design 

6 

2 

Code  &  Unit  Test 

Code  &  Unit  Test 

1 

1 

SI  Test 

Requirements 

29 

50 

SI  Test 

Design 

26 

10 

SI  Test 

Code  &  Unit  Test 

20 

5 

SI  Test 

SI  Test 

1 

1 

Integration 

Requirements 

74 

130 

Integration 

Design 

71 

26 

Integration 

Code  &  Unit  Test 

65 

13 

Integration 

SI  Test 

45 

3 

Integration 

Integration 

1 

1 

Operation 

Requirements 

169 

368 

Operation 

Design 

166 

64 

Operation 

Code  &  Unit  Test 

160 

37 

Operation 

SI  Test 

140 

7 

Operation 

Integration 

95 

3 
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The  derived  formula, 


SLOC  DC  *  SLOC  RC  *  PM  =  CPCD, 

where  SLOC  =  Source  Lines  of  Code 
DC  =  Defect  Cost, 

RC  =  Repair  Cost 
PM  =  Phase  Multiplier 

CPCD  =  Cost  to  Process  and  Repair  Defects 

is  used  to  calculate  the  estimated  costs  to  process  and  correct  defects,  and  the 
avoidance  costs  of  intercepting  the  defects. 

Calculating  the  estimated  costs  to  fix  the  STRs  implemented  by  the 
Developer  using  the  two  different  sets  of  multipliers  yielded  the  following: 

Estimated  Cost  to  Process  and  Correct  Defects 

System  B  System  D 

Boehm  multipliers:  $3.5M  $20M 

Titan  multipliers:  $1.2M  $28M 

As  stated  previously,  the  assumption  is  made  that  had  SIV&V  not 
been  employed,  the  defects  included  in  this  study  would  not  have  been  detected 
until  the  system  had  been  deployed,  thus  resulting  in  dramatically  increased 
costs  to  correct.  As  was  shown  previously,  two  different  sets  of  multipliers  were 
used  to  calculate  the  cost  avoidance  figures: 

Costs  Avoided  by  Intercepting  Defect 


Boehm  multipliers: 
Titan  multipliers: 
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System  B 
$28M 
$  9M 


System  D 
$1 19M 
$203M 


The  deltas  between  the  two  examples  are  very  large  especially  for 
System  D.  As  you  can  see  the  benefits  of  having  an  SIV&V  agent  is  worth  the 
investment.  As  documented  by  Lewis,  Figure  3  below,  latent  requirements  and 
design  errors  can  cost  up  to  36  times  more  to  detect  and  fix  during  test  and 
integration  than  if  caught  in  the  phase  in  which  they  were  generated  (Lewis  1992 
279).  In  retrospect,  if  SIV&V  resources  were  spent  on  all  of  the  development 
phases,  dramatic  reduction  in  costs  could  have  resulted  and  without  question, 
the  worth  of  SIV&V  would  be  justified  as  a  cost  saving  mechanism  and  declared 
“cheap  issuance”  for  the  development  program. 


R  D  C  T 

Phase 


NOMINAL  COST  RATIOS 

D  to  R  =  2.5  :  1 
C  to  R  =  4.9  :  1 
T  to  R  =  36:1 

Error  Cost  Grows  With  Time: 

Requirements  (R)  =  $100 

Design  (D)  =  $250 

Code  &  Unit  Test  (C)  =  $488 

Integration  and  Test  (T)  =  $3568 

SOURCE  :  Independent  Verification  and  Validation: 
A  Life  Cycle  Engineering  Process  for  Quality 

Software.  Lewis,  Robert,  p.  279,  1992 


Figure  3.  Average  Cost  of  Discovering  Errors  (Lewis  1992  279) 


d.  Return  On  Investment  Calculation 

The  growing  interest  to  measure  Return  On  Investment  (ROI)  is 
fueled  by  the  need  to  document  and  measure  improvements  in  both  individual 
and  agent  performance.  The  resulting  ROI  calculation  of  systems  B  and  D  is  as 
follows: 
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Return  On  Investment 


Boehm  multipliers: 
Titan  multipliers: 


System  B 
6.8 
2.1 


System  D 
4.9 
8.8 


Based  on  the  ROI  quoted  in  the  Titan  study  of  an  ROI  of  10,  it  is  not 
unreasonable  at  all  for  the  U.S.  Army’s  Software  Engineering  Directorate  (SED) 
to  state  for  the  systems  analyzed  that  the  ROI  range  is: 

2.1  <  SIV&V  ROI  <  8.8,  with  an  average  of  5.5:1 


If  the  sampling  size  was  increased  for  all  of  the  SED  SIV&V 
programs,  a  more  representative  assessment  could  be  made  to  the  overall  ROI 
that  is  delivered.  This  report  does  conclude,  however,  that  the  ROI  average  for 
SED  SIV&V  efforts  is  likely  within  the  recommended  range.  The  software 
industry  has  yet  to  define  a  recommended  operating  range  for  ROI;  however,  a 
2%  ROI  is  the  projected  break-even  point.  An  ROI  greater  than  or  equal  to  5% 
significantly  justifies  the  effort  and  results  in  cost  improvements,  increased 
quality,  and  enhanced  schedule. 

Another  way  to  look  at  ROI  is  the  differences  between  fixed  and 
variable  costs.  Typically,  costs  for  SIV&V  agents  are  fixed  because  the  SIV&V 
agent  is  focused  on  prevention.  The  developers  cost  are  mainly  variable 
because  the  developer  is  focused  on  detection  and  correction.  If  the  SIV&V 
effort  increases  and  prevents  increasing  numbers  of  defects  then  SIV&V  has 
helped  to  reduce  variable  costs.  As  a  result,  investments  in  SIV&V  should 
continue  as  long  as  the  agents  fixed  costs  remains  below  the  prevented  defects 
cost  (B  +C,  variables  defined  in  ROI  section  above). 
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e.  Development  Schedule  Reduction 

The  Standish  Group  examined  8,380  software  projects  and  found 
that  35%  of  all  software  efforts  were  “challenged,”  16%  were  successful,  and 
31%  cancelled.  The  “challenged”  groups  exceeded  software  delivery  schedules 
by  222%,  were  over  budget  by  189%,  and  were  missing  approximately  39%  of 
the  expected  capabilities  (NASA  1985  2).  Given  this  type  of  track  record,  the 
question  that  many  Program  Managers  (PMs)  ask  is  what  can  they  do  to  improve 
the  software  product  and  reduce  the  development  schedule?  The  answer  is 
SIV&V. 

Software  SIV&V  is  indeed  an  important  aspect  of  developing  quality 
software  (Defense  Acquisition  University  387).  History  has  shown  that  software 
defects  have  delayed  multi-million  dollar  space  launches,  prevented  the  Denver 
airport  opening  for  years,  destroyed  NASA  Missions  (Mars  Climate  Orbiter  and 
Polar  Lander),  killed  service  men  and  women  (e.g.  marines  in  a  helicopter  crash), 
and  shutdown  banking  and  emergency  response  systems. 

A  majority  of  the  time,  SIV&V  does  not  begin  until  the  testing  phase 
of  an  acquisition  program,  where  problems  become  more  noticeable.  Waiting 
until  the  end  to  identify  and  report  problems  only  increases  the  development 
schedule  and  cost.  A  simple  analogy  from  the  auto  industry  can  be  used  here  to 
illustrate  this  issue  within  the  software  world.  “The  mass-producer...”,  in  this 
case  the  software  developer,  “...keeps  the  [assembly]  line  moving  at  all  costs  but 
ends  up  doing  massive  amounts  of  rework  at  the  end,  while  the  lean  producer...”, 
in  this  case  a  PMO  using  SIV&V,  “...spends  more  effort  up  front  correcting 
problems  before  they  multiply  and  ends  up  with  much  less  total  effort  and  higher 
quality  in  the  end”  (Womack  116). 

A  study  published  in  2002  by  the  National  Institute  of  Standards 
and  Technology  (NIST)  estimated  that  software  bugs  are  so  common  that  their 
cost  to  the  American  economy  alone  is  $60  billion  a  year  or  about  0.6%  of  gross 
domestic  product.  According  to  NIST,  80%  of  the  software  development  costs  of 

a  typical  project  are  spent  on  identifying  and  fixing  defects  (Building  A  Better 
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BugTrap  2).  A  bug  which  costs  $1  to  fix  on  the  programmer’s  desktop  costs 
$100  to  fix  once  it  is  incorporated  into  a  complete  program,  and  many  thousands 
of  dollars  if  it  is  identified  only  after  the  software  has  been  deployed  in  the  field 
(Building  A  Better  BugTrap  2). 

SIV&V’s  main  goal  is  to  complement  the  development  effort  by 
reducing  risks  and  identifying  errors,  which  often  lead  to  schedule  slippage.  PMs 
will  receive  the  greatest  payoffs  when  they  utilize  a  SIV&V  agent  for  thorough 
requirements  and  design  verification  aimed  at  preventing  otherwise  costly  errors, 
omissions,  and  inadequacies  from  ever  reaching  the  coding  stages. 

Further,  SIV&V  can  be  very  instrumental  in  finding  errors  during  the 
coding  phase.  On  average,  professional  coders  make  100  to  150  errors  in  every 
thousand  lines  of  code  they  write,  according  to  a  multiyear  study  of  13,000 
programs  by  Humphrey  of  Carnegie  Mellon  (Mann  3). 

Implementing  SIV&V  throughout  the  life  cycle  will  lead  to  better 
software  products,  potentially  reducing  schedules,  and  costs,  in  addition  to 
saving  lives.  If  applied  early  and  effectively,  it  can  help  maintain  balance  in  the 
cost-schedule-quality  equation  throughout  the  development  process  (Callahan). 
If  applied  late  or  reluctantly  (so-called  "11th  hour  V&V"),  it  can  fail  to  have  any 
effect  and  be  cost-ineffective  (Callahan).  Embracing  SIV&V  is  an  endeavor  to 
reduce  the  risks  and  associated  cost-schedule  pressures  inherent  in  the 
development  of  complex  software,  by  changing  the  current  cultural  paradigm  of 
“mass  production”  to  one  of  “lean  production.” 

f.  More  Measurable  Results  of  SIV&V 

In  addition  to  calculating  a  ROI  for  SIV&V  programs  at  SED, 
valuable  data  can  be  gleaned  from  a  close  look  at  a  large  SIV&V  effort  at  SED 
with  20+  years  of  historical  SIV&V  data  available.  This  program,  referred  to  as 
System  D  for  this  report,  has  had  much  success  with  SIV&V,  and  consequently, 
has  had  much  success  as  a  program.  System  D’s  SIV&V  effort  is  about  8-10% 
of  the  Developer’s  yearly  budget,  including  labor,  travel,  and  material.  System 

D’s  SIV&V  is  in-phase,  has  source-level  access,  reports  directly  to  the 
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Government  customer,  and  has  an  enhanced  SIV&V  scope.  System  D’s  SIV&V 
effort  has  been  ongoing  since  the  early  requirements  development  stage  of  the 
system.  These  factors,  combined  with  the  benefits  of  the  SIV&V  being 
performed  by  SED,  which  is  a  Government  Life  Cycle  Center,  have  contributed  to 
the  success  of  the  SIV&V  effort  as  well  as  the  program  itself,  as  demonstrated  by 
the  following  information. 

System  D’s  SIV&V  team  found  and  reported  errors  on  44%  of 
requirements  SLOC  that  were  deemed  completed  by  the  Developer.  Since  the 
SIV&V  is  “in-phase,”  these  requirements  errors  were  able  to  be  corrected  during 
the  requirements  phase.  A  quick  look  at  the  “Study  Multipliers”  in  Table  6  shows 
the  rapid  growth  in  cost  to  correct  that  44%  of  new  requirements  SLOC  as  the 
error  is  propagated  through  the  development  cycle.  At  best,  if  all  of  the  errors 
were  found  and  fixed  in  the  next  phase,  the  cost  would  have  increased  by  a 
factor  of  five.  Near  the  worst  case,  if  all  errors  were  propagated  through 
integration,  but  discovered  before  operational  fielding,  the  cost  would  have 
increased  by  a  factor  of  130.  If  carried  into  operational  fielding,  the  factor  grows 
to  368.  One  must  also  consider  how  many  of  the  requirements  errors  might  have 
gone  undetected  altogether  and  what  the  resulting  cost  would  be  in  dollars, 
schedule,  and  user  benefit.  However,  due  to  the  iterative  nature  of  the 
developer’s  build  cycle,  the  requirements  phase  is  not  closed  until  delivery  of 
requirements  to,  and  return  from,  the  SIV&V  team;  therefore,  the  maximum 
number  of  requirements  errors  can  be  found  and  fixed  during  the  requirements 
phase.  Approximately  44%  of  the  developer’s  new  requirements  SLOC  would 
have  had  errors  that  would  have  been  discovered  and  fixed  at  much  higher  cost 
and  schedule  impact. 

Similarly,  System  D’s  SIV&V  team  found  and  reported  errors  on 
1.2%  of  SLOC  during  the  Design,  Code,  and  Test  phase  that  were  deemed 
completed  by  the  Developer.  Again  looking  at  the  “Study  Multipliers”  in  Table  6  it 
is  easy  to  quickly  spot  the  value  of  finding  and  correcting  these  errors  in  the 
phase  in  which  they  were  generated,  or  quickly  thereafter.  Also,  note  that  this 
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percentage  of  SLOC  with  error  is  artificially  low  due  to  reporting  methods. 
Further  down  the  development  cycle,  System  D’s  SIV&V  team  found  and 
reported  errors  on  0.8%  of  SLOC  during  the  System  Test  and  Integration  Phase. 
Once  more,  this  percentage  is  artificially  low  due  to  reporting  and  tracking 
methods  of  the  developer,  but  provides  insight  into  the  effectiveness  of  SIV&V. 

In  2001,  NASA  conducted  a  study  on  “Developing  Risk-Based 
Financial  Analysis  Tools  and  Techniques  to  Aid  IV&V  Decision-Making.”  The 
study  indicated  that  due  to  SIV&V’s  presence  the  software  product  improved 
tremendously.  By  inserting  SIV&V  into  the  requirements  phase,  the  software 
developer  had  to  rework  22%  of  its  effort  due  to  defects.  Both,  the  design  and 
programming  phases  required  7%  rework.  However,  what  was  most  telling  and 
shocking  was  when  the  SIV&V  effort  was  delayed  until  the  test  and  integration 
phase,  28%  rework  had  to  occur  by  the  developing  organization  to  correct  the 
product. 

There  are  many  other  benefits  the  System  D  SIV&V  team  has 
provided  to  the  Government  customer,  which  cannot  be  easily  tracked  to  a 
metric,  but  give  valuable  payback.  System  D’s  SIV&V  team  is  often  used  as  a 
source  of  expertise  by  the  Government  customer.  Since  the  System  D  SIV&V 
team  was  put  in  place  very  early  in  the  system  development,  and  has  maintained 
an  unusually  low  personnel  turn-over  rate,  both  the  system-level  and  functional- 
level  of  expertise  of  the  SIV&V  team  meets  or  exceeds  the  level  of  expertise  of 
the  developer.  Having  a  second  source  of  expertise,  in  addition  to  the  developer, 
is  valuable  in  three  ways,  1 )  when  the  developer  looses  resources,  2)  when  it  is 
more  efficient  to  have  the  SIV&V  team  work  an  issue  to  prevent  schedule 
impacts  to  the  developer,  and  3)  when  the  SIV&V  team  is  a  cheaper  resource 
than  the  developer. 

System  D’s  SIV&V  team  has  been  relied  upon  heavily  during  times 

of  war  and  when  the  soldier  in  the  field  needs  an  explanation,  since  the  software 

developer  is  typically  no  longer  available,  and  for  special  studies  and 

investigations.  Since  SED,  a  Government  institution  is  System  D’s  SIV&V 
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resource,  the  Army  Evaluation  Command  relies  on  the  SIV&V  team  for  testing 
requirements  that  are  outside  of  their  resource  limitations.  System  D’s  SIV&V 
team  also  represents  the  Government  customer  on  the  System  Safety  Analysis 
Board. 

So  when  taking  a  close  look  at  an  example  of  SIV&V  in  action,  the 
value  of  SIV&V,  when  implemented  properly,  is  measurable  in  both  cost  and 
intrinsic  value,  and  both  values  are  large.  System  D’s  example  of  success  also 
demonstrates  several  crucial  keys  to  successful  SIV&V.  Historical  data  from 
many  sources  demonstrates  that  SIV&V  does,  in  general,  have  a  positive  ROI. 
Combining  “classical”  SIV&V  with  lessons  learned  and  crucial  keys  from  the  rich 
history  at  SED  can  drive  the  ROI  to  its  maximum  value. 

g.  Examples  With  and  Without  SIV&V 

(1)  Ariane  5  Without  SIV&V.  On  June  4,  1996,  the 
maiden  flight  of  the  European  Ariane  5  launcher  crashed  about  40  seconds  after 
takeoff.  Media  reports  indicated  that  the  cost  of  this  lose  was  half  a  billion  dollars 
--  uninsured  (Knutson;  Missile  Defense  Agency  24). 


Figure  4.  Ariane  5  Crash  (Knutson;  Missile  Defense  Agency  24) 

Ariane-5  was  the  newest  in  a  family  of  rockets  designed  to 
carry  satellites  into  orbit.  On  its  maiden  launch  on  June  4,  1996,  it  flew  for  just 
under  40  seconds  before  self-destructing,  destroying  the  rocket  and  its  payload 
of  four  satellites  (Knutson;  Missile  Defense  Agency  24). 
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After  the  incident,  Ariane  immediately  set  up  a  Board  of 
Inquiry  to  conduct  a  thorough  investigation  to  discover  the  root  cause  of  the 
accident  (Knutson;  Missile  Defense  Agency  26). 

The  core  problem  in  the  Ariane  failure  was  incorrect  software 
reuse.  A  critical  piece  of  software  was  reused  from  the  Ariane-4  system,  but 
behaved  differently  in  the  Ariane-5  because  of  differences  in  the  operational 
parameters  of  the  two  rockets.  One  of  the  important  lessons  from  the  Ariane-5 
failure  is  that  the  quality  of  a  device's  software  must  be  considered  in  the  context 
of  the  entire  system.  It  is  an  important  lesson  to  keep  in  mind  as  software  reuse 
continues  to  be  an  important  trend  in  software  engineering.  If  SIV&V  were 
utilized  these  abnormal  conditions  could  have  been  avoided  (Knutson;  Missile 
Defense  Agency  26). 

The  Ariane  failure  was  highly  publicized  and  documented. 

(2)  NASA  With  SIV&V. 


Cost  of  Failure  is  the 
Rationale  for  NASA  Support 

This  Document  Is  Uncontrolled  When  Printed.  I  to  SW  IV&V 

Check  the  NASA  Online  Directives  Information  System  (NODIS)  Library 
to  verify  that  this  is  the  correct  version  before  use: 
http  ://no  dis .  hq .  nas  a.  gov/Library/Dir  e  ctive  s/NAS  A-WIDE/c  ontents .  html 


Responsible  Office:  Q  /  Office  of  Safety  and  Mission  Assurance 
Subject:  Software  Independent  Verification  and  Validation  (IV&V)  Policy 
1.  POLICY 


MAS A  will  conduct  Independent  Verification  and  Validation  (IVsV)  based  on 
Che  cost,  size,  complexity,  life  span,  risk,  and  consequences  of  failure. 
Specifically,  NASA  will: 

a.  Establish  and  apply  criteria,  tools,  and  methodology  to  evaluate  and 
assess  software  risk  for  the  purpose  of  identifying  the  appropriate  level 
of  IVSV. 

b.  For  programs  and  projects  governed  by  NPG  7120. 5A,  task  the  NASA  IV £V 
Facility  in  Fairmont,  W,  to  manage  the  performance  of  all  IV £V  for 
software  identified  per  the  established  criteria,  and  for  any  other  safety 
critical  software  (as  defined  in  NASA-STD-8719. 13A) . 

c.  Require  programs  and  projects  governed  by  NPG  7120. 5A  to  determine  the 
level  of  IV SV  to  be  performed  with  the  explicit  involvement  of  the  NASA 
IVCV  Facility. 

d.  Require  NASA  programs  and  projects  that  contain  mission  or  safety 
critical  software  to  document  decisions  concerning  the  use  of  IVfiV. 
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Figure  5.  NASA  SIV&V  Policy  (Missile  Defense  Agency  23) 
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h.  DoD  SIV&V Examples: 

(1)  AEGIS  Without/With  SIV&V.  Recent  headlines  stated 
“AEGIS  SHIPS  USS  VICKSBURG  and  USS  HUE  CITY  Docked  After  Contractors 
Have  ‘Free-Reign’  of  Software  -  Deemed  Not  Seaworthy  Until  SIV&V  is 
Reinstated”  (Missile  Defense  Agency  27),  and  “Software  Glitches  Leave  Navy 
Smart  Ship  Dead  In  The  Water”  (Slabodkin). 

The  cost  for  NSWCDD's  Aegis  Baseline  SIV&V  process 
varies  significantly  based  on  the  size  and  complexity  of  the  functional  capability 
that  is  being  developed.  A  software  development  effort  such  as  Baseline  6 
Phase  3  costs  as  much  as  $50M  LOE  spread  over  the  design,  code,  integration 
and  test  phases  in  a  4-5  year  span  for  the  SPY-1  (Radar)  element  alone. 
Numerous  significant  issues  and  problems  were  identified,  investigated,  and 
resolved  prior  to  major  milestones  as  a  result  of  NSWCDD's  SIV&V  involvement. 
Considering  that  SPY-1  Baseline  6  Phase  3  cost  well  over  $500M  to  develop  and 
field,  $50M  for  SIV&V  was  a  small  price  to  pay  (Missile  Defense  Agency  27). 

The  yearly  cost  NSWCDD  applies  to  SPY-1  SIV&V  for  SW 
development  is  about  $10M.  This  includes  personnel  to  support  design  reviews, 
code  inspections,  unit  testing,  test  procedure  development,  formal  element 
testing,  formal  system  level  testing,  SW  configuration  management,  SW  quality 
assurance,  SW  documentation,  training,  and  facility  operations  (Missile  Defense 
Agency  27). 

(2)  THAAD  Without/With  SIV&V.  After  Three  Repeated 
Flight  Test  Failures  (Dem/Val),  THAAD  Project  Office  requested  SIV&V  to  test  all 
software  changes  to  verify  correct  software  implementation. 
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Figure  6.  THAAD  Electronics  Box  (U.S.  Army  28) 


The  THAAD  missile  SIV&V  effort  identified  10  percent 
(approximately  200)  of  all  avionics  software  problems. 

•  Many  critical;  fixes  had  to  occur  before  flight  tests 
resumed 

•  SIV&V  found  these  problems  in  released  (tested)  code 

•  SIV&V  testing  helped  get  the  THAAD  program  on  track 

i.  Commercial  Examples:  No  SIV&V  Agent 

Utilized 

(1)  FAA.  The  Federal  Aviation  Administration  (FAA) 
encountered  a  software  glitch;  software  patches  were  dispatched  to  Boston  and 
other  airports  to  enhance  the  ground-based  radar  systems.  The  new  software 
failure  was  noted  when  the  system  could  not  see  two  planes  approaching  each 
other  on  the  runway,  which  was  a  failure  caused  by  the  patch.  The  FAA  reported 
in  another  incident  that  a  backup  system  that  was  designed  to  handle  planned 
server  downtime  resulted  in  a  three  hour  loss  of  air  traffic  control  communications 
between  800  plus  airplanes  (Entries  Hall  of  Shame). 

(2)  AT&T.  American  Telephone  and  Telegraph  (AT&T) 
experienced  a  billion  dollar  failure  in  January  1990,  when  a  software  failure  took 
down  the  entire  United  States  (US)  long  distance  telephone  network  for  nine 
hours  (AT&T  Wireless). 
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(3)  US  Airways.  US  Airways  experienced  a  software 
failure  in  April  2005,  when  the  ticketing  system  issued  incorrect  fares  for  several 
hours.  During  this  occurrence  some  tickets  were  sold  for  under  $2.00.  The 
airline  honored  the  reduced  fares  as  a  gesture  of  good  faith,  but  lost  a  substantial 
amount  due  to  the  failure  (Entries  Hall  of  Shame). 

(4)  Toyota.  In  October  2005,  the  Toyota  Motor  Company 
recalled  more  than  75,000  Toyota  Prius-hybrids  due  to  a  software  failure  that 
may  have  shut  down  the  engine.  Toyota  quickly  implemented  the  recall  which 
avoided  having  the  defect  become  permanently  associated  with  the  vehicle  line 
or  with  hybrid  safety  (Entries  Hall  of  Shame). 

C.  APPLYING  SIV&V  IN  THE  CYCLE  PHASE 

The  following  sections  are  primarily  based  on  an  IV&V  process  chart 
authored  by  R.  O.  Lewis  (Lewis  1998). 

1.  Introduction 

Software  Independent  Verification  and  Validation  (SIV&V)  activities 
(notional  list  of  activities  shown  in  Figure  7  below)  are  used  to  assess  risk  and 
the  quality  of  software  throughout  the  development  process  and  are  performed  at 
the  unit,  module,  integration,  and  system  levels.  SIV&V  activities  should  begin 
early  in  the  software  development  process  to  assure  consistency  between 
product  specifications  and  requirements,  design,  implementation  and  testing. 
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Figure  1. 1V&  V  Activities. 


Figure  7.  Notional  SIV&V  Activities  (Walters) 

Independent  reviews  are  conducted  during  and  at  the  end  of  each  phase 
of  the  life  cycle  to  determine  whether  established  requirements,  design  concepts, 
and  specifications  have  been  met.  The  customer  relies  on  the  Independent 
agent  to  provide  a  go  or  no-go  decision  on  whether  to  proceed  to  the  next  step  or 
start  the  process  over.  The  following  sections  will  highlight  some  of  the  different 
activities  that  are  required  of  SIV&V  in  each  cycle  phase. 

2.  Concept  Definition  Phase 

Concept  Development  is  typically  the  first  phase  of  a  major  system 
development  program.  Normally,  this  phase  includes  competition  among 
contractors  who  offer  different  solutions  to  some  basic  need,  in  which  case  they 
often  either  produce  prototypes  of  their  designs  or  conduct  studies  and  analyses. 
Although,  the  process  may  vary  significantly,  programs  that  have  a  formal  SIV&V 
effort  at  this  point  in  the  life  cycle,  invariably  end  up  improving  the  System 


Specification  (SS),  analyze  the  external  interface  requirements  and  associated 
inputs,  and  examine  the  feasibility  and  adequacy  of  each  competing  design. 
SIV&V  reviews  all  available  input  materials  to  enhance  its  program 
understanding  as  indicated  by  the  following  tasks  (Lewis  1998): 

•  Initial  SIV&V  activities  focus  on  evaluation  of  the  mission  needs, 
external  system  requirements,  external  interfaces,  and  overall  program 
planning  data.  This  should  include  a  survey  of  the  users  requirements, 
examination  of  make-buy  decisions,  assessment  of  technology  drivers, 
and  evaluation  of  the  operational  concept. 

•  The  various  program  plans  are  assessed  for  consistency, 
completeness,  and  correctness  to  ensure  essential  aspects  of  the 
conceptual  system  are  addressed.  These  include  but  are  not  limited 
to:  interface  control,  configuration  management,  safety,  risk 
management,  master  program  plan  schedules,  integration,  resource 
control,  environment,  and  planning  for  reaching  operational  capability. 

•  SIV&V  evaluates  the  system  concepts  against  the  requirements 
mission  and  user  needs.  This  information  is  used  to  assist  in  the 
evaluation  of  the  Technical  Requirements  Document  (TRD),  system 
requirements,  and  drafts  of  the  SS  or  Prime  Development  Specification 
(PIDS)  produced  and  controlled  by  the  customer. 

•  A  System  Requirements  Review  (SRR)  covers  each  draft  release  of 
the  System  Specification.  There  are  often  several  SRRs  held  to  firm 
up  the  requirements. 

•  The  SIV&V  Plan  is  drafted  as  early  as  possible,  but  usually  has  to 
await  the  generation  of  the  contractor’s  Software  Development  Plan 
(SDP)  for  completion;  because,  much  essential  information  is  still 
missing. 

•  As  the  program  requirements  become  stable,  SIV&V  examines 
program  feasibility  and  completeness,  allocation  of  requirements  to 
major  subsystem  elements,  and  supports  the  initial  conversion  of  the 
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critical  functions  and  requirements  into  a  formal  listing  or  requirements 
trace  database.  Other  products,  such  as  the  discrepancy  logs  are 
developed  and  captured  during  this  period. 

•  As  the  SIV&V  team  identifies  potential  problems  and  issues,  reports 
are  submitted  to  the  customer’s  Configuration  Management  (CM)  point 
or  designated  customer  representative.  These  problems  or  anomalies 
are  typically  reported  and  summarized  in  status  reports. 

•  Engineering  analysis  is  used  to  evaluate  the  evolving  systems 
concepts.  If  the  system  is  extremely  complex,  high  level  modeling  and 
simulation  are  often  used  to  evaluate  the  completeness  and  feasibility 
of  competing  system  designs. 

•  SIV&V  identifies  inconsistencies  and  shortcomings  in  concepts, 
technology  drivers,  make-buy  decisions,  documentation,  trades 
studies,  and  the  evolving  System  Specification.  Special  emphasis  is 
placed  on  critical  software  issues  including:  user  and  performance 
requirements,  operational  feasibility,  modeling  and  prototyping, 
hardware  performance  needs,  safety  issues,  and  system  and  software 
risks  and  their  mitigation. 
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SIV&V  Battle  Rhythm 
(Concept  Development  Phase) 


Input  -  Entrance  Criteria 

-  Mission  Needs  Statement 

-  Operational  Requirements  Document 

-  Initial  Mission  Planning 

-  External  System  Requirements 

-  Overall  Program  Planning  Data 

-  Identify  Interoperability  Constraints 

-  Make-Buy  Decisions 

-  Survey  of  User  Needs 

-  Initial  System  Concepts 

-  Feasibility  Study  Reports 


Output  -  Exit  Criteria 

-  System  Specification  or  Prime  Item 
Development  Specification  (PIDS)  Draft 
Complete 

-  Program  Management  Plan 

-  Draft  Preliminary  System  Design  Draft 

-  Complete  System  Concept  Studies 

-  Initial  System  Engineering  Trade 

-  Perform  Special  Studies 

-  Define  Hi-Level  System  Architecture 

-  Verified  Concept  Study 

-  Review  System  Engineering  Management 
Plan 

-  Reports  of  System  Requirements 
Specifications 


Figure  8.  SIV&V  Battle  Rhythm  Concept  Development  Phase  (Lewis  1998) 


3.  Requirements  Phase 

“During  the  requirements  phase,  the  user  tries  to  articulate  a  concept  of 
expected  system  function  and  performance  into  concrete  detail”  (Department  of 
Air  Force  2-20).  SIV&V  activities  during  the  requirements  phase  include  analysis 
of  the  software  and  hardware  requirements  to  determine  if  the  software 
engineers  have  translated  user  definitions  into  realistic  and  achievable 
specifications.  Similarly,  SIV&V  determines  if  the  established  requirements  are 
testable  and  capable  of  being  satisfied.  The  requirements  phase  is  a  segment 
that  first  concentrates  on  system  requirements,  and  then  software  requirements. 
The  activity  verifies  that  the  software  requirements  have  been  prepared  in 
accordance  with  applicable  standards,  and  evaluates  the  progress  of  the 
requirements  toward  achieving  an  operational  system.  Requirements  Analysis 
identifies  critical  risks  and  potential  process  and  product  improvements. 
Potential  risks  due  to  defects  in  the  requirements  are  identified  and  addressed 
through  discussions  with  the  developer.  High  priority  risks  are  analyzed  to 
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identify  potential  process  and  product  improvements.  This  multi-part 
requirements  phase  is  in  concert  with  most  approved  development  standards 
and  typically  encompasses  the  following  tasks  (Lewis  1998): 

•  During  concurrent  engineering,  hardware  and  software  requirements 
are  synthesized  to  include  numerous  trade-offs  and  performance 
considerations.  Hardware  benchmarks  are  used  to  measure 
performance  using  typical  applications  and  are  verified  by  SIV&V. 

•  This  phase  addresses  the  following  basic  verification  criteria,  namely 
system  and  software  requirements  verification,  and  evaluation  of  the 
initial  test  plan  and  the  SIV&V  test  plan.  This  phased  approach 
accomplishes  additional  strategic  tasks  more  or  less  as  follows: 

o  Inputs  from  Concept  Development  Phase  plus  those  listed 
above. 

o  Initial  steps  involve  verifying  the  operational,  functional, 
performance,  and  program  requirements,  and  assessing  and 
racking  the  critical  functions  including  the  use  of  SIV&V 
requirements  tracking  database  or  tool. 

o  Plans  and  planning  factors  are  evaluated  to  ensure  that  SIV&V 
works  in  lock  step  with  the  development  team. 

o  Evaluate  the  engineering  trades  to  ensure  feasibility  and 
completeness  of  the  requirements  documents,  which  can  be 
measured  quantitatively. 

o  Examine  anticipated  behaviors  and  performance  needs  of  the 
system  and  its  software  to  develop  initial  Measures  of 
Performance  (MOPs). 

•  The  System  Specification  is  verified  and  analyzed  for  completeness, 
consistency,  testability,  correctness,  understandability,  and  feasibility. 
If  a  System  Segment  Design  Document  (SSDD)  is  produced,  it  is 
verified  as  well. 

•  Prototypes  or  models  of  the  system  architecture  are  evaluated. 

•  SIV&V  participates  in  the  System  Design  Review  (SDR)  and  Critical 
Design  Reviews  (CDR). 

•  SIV&V  assesses  the  development  standards  and  guidelines  used. 
This  includes  review  of  the  draft  and  final  Software  Development  Plan 
(SDP). 
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•  During  this  time,  program  data  collection  begins  in  earnest  and  feeds 
the  SIV&V  status  report.  Early  metrics  typically  include  requirements 
stability  and  adequacy.  Risk  tracking  begins  and  the  highest  risks 
items  are  routinely  monitored.  SIV&V  program  status  and  performance 
metrics  augment  the  status  reporting. 

•  Following  the  SDR  and  the  functional  base-lining  of  the  System 
Specification,  SIV&V  emphasis  shifts  to  software  requirements  and  to 
the  extent  necessary,  hardware  requirements. 

•  As  the  interface  definitions  mature,  SIV&V  reviews  and  evaluates  the 
Interface  Requirements  Specification  (IRS)  and  Interface  Control 
Document  (ICD).  Agreement  between  these  documents  and  the 
SSDD  is  vital. 

•  The  allocation  of  functions  and  requirements  to  Hardware 
Configuration  Items  (HWCIs)  and  Software  Items  (Sis)  is  verified  to 
provide  the  foundation  for  in-depth  verification  of  each  SRS  for  each 
SI. 

•  SIV&V  participates  in  the  Software  Specification  Review  (SSR)  to 
ensure  that  each  CSCI  is  ready  to  transition  into  the  architectural 
design. 

•  It  is  here  that  analytical  methods,  tools,  and  techniques  are  used 
effectively  to  evaluate  and  verify  the  requirements. 

•  Hardware,  support  software,  and  tools  selected  for  the  Software 
Engineering  Environment  (SEE)  are  evaluated  and  recommendations 
are  provided. 

•  Special  emphasis  is  placed  on  critical  software  issues. 
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SIV&V  Battle  Rhythm 
(Requirements  Phase) 


Input  -  Entrance  Criteria 


-  Concept  Study  Reports 

-  High-Level  Architecture  of  System 

-  Established  Concepts  of  Need,  Requirements, 
and  Objectives 

-  Identified  Technology  Requirements 

-  Initial  Risk  Assessment 

-  Initiate  Configuration  Management  (CM) 
Controls 

-  System  Requirements  Trade  Studies 


Output  -  Exit  Criteria 

-  Specifications  Tree  &  Work  Breakdown 
Structure  (WBS)  Final 

-  Approved  System  Specification  (SS) 

-  Approved  System/Segment  Design 
Documentation  (SSDD) 

-  Any  Prototype  Evaluated 

-  Approved  CSCI  Software  Requirement 
Specification  (SRS) 

-  Approved  SEMP 

-  Interface  Requirements  Specification  (IRS) 
Complete 

-  Critical  requirements  Identified 

-  Results  of  Systems  Design  Review  (SDR) 
and  Software  Specification  Review  (SSR) 


Figure  9.  SIV&V  Battle  Rhythm  Requirements  Phase  (Lewis  1998) 

4.  Design  Phase 

The  Design  Phase  activity  is  conducted  by  reviewing  the  software  design 
products  (e.g.,  object  models,  sequence  diagrams,  algorithm  descriptions,  design 
specifications,  etc.)  of  each  software  baseline  build  and  evaluating  the  products 
for  adherence  to  software  requirements.  The  design  products  (e.g.  Software 
Design  Documents  (SDDs)]  are  also  analyzed  for  consistency,  completeness, 
and  correctness  with  respect  to  the  system,  software,  and  interface 
specifications.  Software  problems  identified  in  the  Design  Analysis  activity  are 
documented  as  risks,  and  comments  are  provided  to  the  developer  for 
adjudication.  Typically,  the  Design  Phase  is  multifaceted,  meaning  that  it 
consists  of  a  Preliminary  Design  Phase  or  Architectural  Design  Phase  and  a 
Detailed  Design  Phase.  These  facets  and  SIV&Vs  input/output  processes  are 
discussed  below  (Lewis  1998). 
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a.  Architecture  Design 

Architectural  design  verification  gives  attention  to  the  soundness 
and  completeness  of  the  software  design,  interface  design,  and  traceability  of  the 
requirements  into  the  design.  The  following  tasks  are  typical  of  this  phase  (Lewis 
1998): 

•  This  process  consists  of  a  number  of  steps  that  evaluate  the 
allocation  of  functions  and  capabilities  especially  centered  on 
critical  functions,  a  thorough  examination  of  the  architectural 
design  and  its  documentation  in  the  preliminary  Software 
Design  Document  (SDD),  and  verification  of  the  system 
architecture. 

•  The  SIV&V  team  evaluates  the  fidelity  of  the  design  to  ensure 
adequacy  of  algorithms  given  the  performance  expected  from 
the  elected  platform.  This  is  accompanied  by  analysis  of  the 
operating  timeline  estimates. 

•  The  architectural  design  is  evaluated  for  its  completeness, 
modularity,  efficiency,  complexity,  stability,  and 
understandability. 

•  The  interface  requirements  are  tracked  from  the  Interface 
Requirements  Specifications  (IRS)  into  the  Sis  and  HWCIs  and 
into  the  Interface  Control  Document  (ICD).  The  interfaces  are 
evaluated  for  completeness,  consistency,  correctness,  and 
ability  to  implement. 

•  Externally  supplied  data  is  assessed  for  adequacy  and 
completeness,  and  is  certified,  when  necessary. 

•  Internal  data  is  analyzed  using  data  flows  generated  by  selected 
software  tools.  A  data  dictionary  is  produced  by  the  developer, 
which  is  evaluated  by  SIV&V  for  accuracy  and  completeness. 

•  SIV&V  proven  methods,  tools,  and  techniques  are  selected  and 
used  for  analysis  and  evaluation. 
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•  Test  requirements  are  extracted  and  traced  to  the  developer’s 
Software  and  Hardware  Test  Plans  (STP  and  HTP).  This 
ensures  adequacy  of  the  test  plans  and  also  provides  data  for 
development  of  the  SIV&V  Test  Plans  when  independent  testing 
is  required.  Many  systems  today  are  networked  and  require 
extensive  ‘distributed’  test  environments;  it  is  increasingly 
difficult  and  cost  prohibitive  to  attempt  to  exactly  replicate  these 
facilities  for  SIV&V.  In  such  cases,  ‘dual-use’  testing  is 
encouraged  in  which  SIV&V  and  developers  share  facilities  and 
conduct  joint  testing. 

•  User  requirements  are  verified,  and  safety  factors  and  the 
Failure  Mode  Effects  Analysis  (FMEA)  are  evaluated. 

•  Change  requests  to  the  design  and  requirements  are  tracked, 
coordinated,  and  evaluated  for  impact  to  the  design 
documentation. 

•  The  architectural  design  portion  of  the  SDD  is  thoroughly 
evaluated  for  completeness,  consistency,  traceability, 
correctness,  implement-ability,  and  testability. 

•  SIV&V  participates  in  the  Architectural  or  Preliminary  Design 
Review  (ADR  or  PDR). 

•  Human  engineering  factors,  command  and  control  features,  and 
functions  are  evaluated  to  ensure  adequate  user  and  operator 
interaction. 

•  All  critical  functions  are  traced,  verified,  and  documented  in  the 
SIV&V  requirements  tracing  database. 

•  The  issue  Tracking  Log  tracks  the  status  of  all  essential  open 
issues. 

•  The  SIV&V  Test  Plan  is  drafted  and  may  become  an  extension 
of  the  developer’s  Software  Test  Plan  (STP),  if  dual-use  (joint) 
testing  is  conducted. 
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•  Development  changes  that  occur  during  this  phase  are 
immediately  fed  into  the  appropriate  verification  activity  and 
assessed  for  their  impact. 

SIV&V  Battle  Rhythm 
(Architectural  Design  Phase) 

Input  -  Entrance  Criteria 


n=> 


-  Draft  Data  Requirements 

-  System  Performance  Specified 

-  System  Level  Trade  Studies  Complete 

-  Approved  Systems  Specifications  (SS)  or  Prime 

-  Item  Development  Specification  (PIDS)  and 
System  Segment  Design  Document  (SSDD) 

-  Identify  Critical  Functions 

-  Approved  Software  Requirements  Specification 
(SRSs) 

-  Approved  Interface  Requirements  Specification) 
IRS(s) 

-  Functional  Allocations  Complete 

-  Draft  Architecture  Language 


Output  -  Exit  Criteria 

-  Approved  Interface  Control  Document 
(ICD) 

-  Preliminary  Software  Design  Documents 
(SDD)  Complete 

-  Critical  Functions  Listed 

-  Architecture  Complete 

-  Operating  Timelines 

-  Behavioral  Analysis 

-  Initial  Timing  &  Sizing  Estimates 

-  Safety  Factors  and  Failure  Mode  Effects 
Analysis  (FEMA)  Assessed 

-  Risk  Management  Initiated 

-  Preliminary  Design  Review 

(PDR)/ Architectural  Design  Review  (ADR) 
Issues  Tracked  to  Closure 


Figure  10.  SIV&V  Battle  Rhythm  Architectural  Design  Phase  (Lewis  1998) 

b.  Detailed  Design 

Detailed  design  verification  examines  the  detailed  design  of  the 
software  for  completeness,  consistency,  logical  and  functional  correctness, 
implement-ability,  and  feasibility  and  typically  addresses  the  following  tasks 
(Lewis  1998): 

•  SIV&V  examines  and  assesses  the  design  for  efficiency, 
modularity,  fidelity,  complexity,  testability,  clarity,  flexibility,  and 
stability. 

•  Metrics  and  performance  indicators  are  collected,  assessed, 
and  reported  back  to  the  customer  on  program  maturity  and 
potential  delinquencies. 
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•  SIV&V  participates  in  design  inspections  hosted  by  the 
developer  and  customer  to  evaluate  the  thoroughness  and 
discipline  of  the  design  team;  this  is  especially  true  when 
integrated  product  teams  are  used. 

•  In  view  of  the  fact  that  detailed  design  of  complex  systems  is 
invariably  supported  by  software  design  tools,  SIV&V  evaluates 
them  and  identifies,  selects,  and  uses  portions  of  the 
developer’s  tool  set  to  ensure  repeatability;  additionally, 
specially  chosen  complementary  SIV&V  tools  are  used  to 
enhance  analysis  and  error  detection;  and  further  investigate 
suspect  designs. 

•  SIV&V  performs  several  verification  activities  in  parallel,  that 
include  hardware/software  mapping;  verification  of  key 
algorithms;  analysis  of  control  flow,  schema,  and  behaviors;  and 
detailed  timing  and  sizing  analysis.  Algorithms  are  selected 
based  on  criticality,  complexity,  and  performance,  and  are 
thoroughly  evaluated,  often  by  coding  and  executing  them  on  a 
tool-bearing  host  to  determine  their  accuracy,  performance,  and 
suitability. 

•  The  Interface  Design  Document  (IDD)  and  Interface  Control 
Document  (ICD)  are  evaluated  for  consistency,  completeness, 
and  accuracy.  This  includes  hardware  interface  assessment  to 
ensure  integrity  of  the  design. 

•  SIV&V  verifies  the  developer’s  Software  Test  Descriptions 
(STDs)  and  Software  Test  Plans  (STPs).  This  process  ensures 
adequacy  of  both  the  development  and  SIV&V  test  programs  to 
fully  verify  all  critical  requirements  and  functions.  When  dual- 
use  (joint)  testing  is  employed,  SIV&V  feeds  its  discrete  test 
requirements,  test  cases,  and  data  collection  needs  to  the 
developer  for  inclusion  in  its  testing.  This  technique  is  used 


52 


when  it  is  impractical,  too  expensive,  or  impossible  to  precisely 
replicate  the  development  configuration. 

•  SIV&V  thoroughly  evaluates  the  user  and  operator  interfaces 
and  command  and  control  interactions,  and  often  participates  in 
safety  assessments  of  the  software. 

•  Execution  time  budgets  and  task  schedules  are  assessed  to 
estimate  the  adequacy  of  operating  margins. 

•  SIV&V  verifies  the  completed  SDDs  based  on  virtually  all 
detailed  design  activities  discussed  in  this  section. 

•  Continue  verifying  and  tracking  critical  functions  using  the 
SIV&V  requirements  tracking  database  and  critical  function  list. 

•  SIV&V  evaluates  the  visualization  and  the  Graphical  User 
Interfaces  (GUIs)  proposed  for  use  or  prototyped  for  the 
system — screens,  human  machine  interfaces,  controls,  use  of 
color,  mimics,  and  so  forth. 

•  SIV&V  participates  in  hardware  evaluation  and  design  to  benefit 
the  software  design  verification.  A  strong  understanding  of  the 
hardware  is  essential  to  software  SIV&V. 

SIV&V  Battle  Rhythm 
(Detailed  Design  Phase) 

Input  -  Entrance  Criteria 


in=> 


Figure  1 1 .  SIV&V  Battle  Rhythm  Detailed  Design  Phase  (Lewis  1998) 


-Architectural  Design  Complete  per  Software  Design 
Document  (SDD)s  and  Tool  Output 

-  Preliminary  Timing  and  Sizing  Data 

-  Design  Trade  Studies 

-  Commercial  Off-The-Shelf  (COTS)/GOTS  Selection 

-  Input/Output  (I/O)  Data  Sources/Definitions 

-  Development  Environment  Software  Engineering 
Environment  (SEE)  Established 

-  Draft  Interface  Design 

-  External  Design  Constraints 

-  Functional  and  Performance  Requirements  Database 
(DB) 


Output  -  Exit  Criteria 

-  Detailed  Designs  Complete  per  Critical 
Design  Review  (CDR)/Detailed  Design 
Review  (DDR) 

-  Agreement  on  Critical  Functions 

-  Complete  Software  Design  Documents 
(SDDs)  for  each  Computer  Software 
Configuration  Item  (CSCI)/Software  Item  (SI) 

-  Complete  Interface  Design  Document  (IDD) 

-  Complete  Database  Design  Document 
(DBDD) 

-  Verified  Operating  Timelines 

-  Risk  Management  Performed 

-  Closure  on  All  Critical  Design  Review 
(CDR)/(Detailed  Design  Review  (DDR) 

Issues 

-  Approval  for  Coding 
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5.  Code  and  Development  Testing  Phase 

Code  verification  is  commonly  a  three-phase  activity  used  with  most 
development  practices.  This  forms  a  suitable  way  of  viewing  the  evolution  of 
code  from  the  Computer  Software  Unit  (CSU)  (smallest  compliable  entities),  to 
the  Computer  Software  Component  (CSC)  (collections  of  units  that  have 
common  or  supporting  functions),  to  Software  Item  (SI)  level  and  supports  the 
testing.  The  Institute  of  Electrical  and  Electronics  Engineers  (IEEE)  J-STD-016 
and  others  handle  these  steps  as  two  parts,  in  which  “Software  Units  (SUs)” 
comprise  everything  below  the  SI  level.  Despite  the  number  of  discrete  levels  of 
coding  the  SW  goes  through,  this  pattern  depicts  the  growth  from  small  chunks  of 
code  to  components  (tasks  and  packages)  and  finally  to  complete  Sis.  The 
following  tasks  are  typical  of  this  phase  (Lewis  1998): 

•  SIV&V  typically  begins  code  verification  at  the  level  where  execution  of 
the  code  can  occur.  It  is  often  too  hard  to  force  a  CSU  to  execute  and 
do  something  representative,  so  this  level  of  verification  is  usually 
avoided  as  it  is  too  costly  and  time  consuming.  The  converse  is  true  at 
the  CSC,  package,  and  SI  level.  Beginning  with  CSC,  the  code  can 
usually  perform  one  or  more  complete  tasks,  and  therefore,  can  be 
evaluated  and  verified  in  a  meaningful  way.  There  are  many  variations 
to  this  approach  resulting  from  differences  in  software  languages, 
developer  preferences,  maturity  of  the  software,  stability  of  the 
requirements,  and  the  like. 

•  One  of  SIV&V’s  favored  approaches  is  to  use  a  code  analyzer  (i.e. 
McCabe,  FlexeLint,  and  Vector  Cast)  on  CSCs  and  Sis.  These  tools 
find  a  large  number  of  the  embedded  coding  errors,  and  dead  code, 
inspect  the  code  for  standards  and  rules  violations,  uncover  bad 
coding  practices  defined  by  industry  (i.e.  Carnegie  Mellon  Software 
Engineering  Institute),  and  semantic  errors.  These  tools  are  able  to 
very  quickly  calculate  the  complexity  of  code,  and  help  point  the  SIV&V 
analyst  to  the  hot  spots  to  make  assessments  and  look  for  possible 
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problems.  These  tools  can  support  static  and  dynamic  execution  of 
software  on  most  platforms.  A  real  plus  is  that  these  tools  are  very  fast 
and  yield  consistent  results  after  repeated  runs,  something  human 
analysts  cannot  always  do. 

•  In  addition  to  the  code  integrity  checks  described,  SIV&V  completes 
the  verification  of  the  SDDs,  IDD,  ICD,  and  the  design  tools’  outputs  to 
ensure  that  the  designs,  databases,  and  interface  documents  match 
the  code. 

•  Often  times,  SIV&V  participates  in  code  inspection,  peer  reviews,  and 
walk-throughs. 

•  SIV&V  ensures  that  the  standards  and  coding  practices  described  in 
the  developer’s  Software  Development  Plan  (SDP)  are  being  followed. 

•  SIV&V  evaluates  the  adequacy  of  the  developer’s  test  procedures  and 
the  test  facilities  and  environment.  This  includes  simulations,  drivers, 
simulators,  and  database  analysis  tools. 

•  SIV&V  then  generates  test  procedures  and  conducts  independent 
testing  (i.e.  “white  box/black  box”  testing)  to  complement  testing  being 
performed  by  the  developer,  or  as  previously  described,  may  input  its 
requirements  into  the  developer’s  STD  for  dual  use  testing. 

•  SIV&V  continues  tracking  and  focusing  on  the  critical  functions  to 
ensure  the  highest  possible  degree  of  safety,  reliability,  and  user  and 
mission  responsiveness. 

•  This  phase  provides  an  opportunity  to  verify  the  networking  for  the  test 
environment  to  support  SIV&V.  This  is  especially  important  when  the 
developer  and  SIV&V  teams  are  separated  geographically. 

•  Software  change  activity  is  typically  heaviest  during  this  phase  and  the 
next  phase,  stressing  the  Configuration  Management  (CM),  problem 
tracking,  and  reporting  systems. 
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SIV&V  Battle  Rhythm 
(Code  &  Development  Testing 
Phase) 


Input  -  Entrance  Criteria 


-  Critical  Design  Review  (CDR)/Detailed  Design 
Review  (DDR) 

-  Software  Design  Documents  (SDDs)  Complete 

-  Database  Design  Document  (DBDD)  Complete 

-  Interface  Design  Review  (IDD)  Complete 

-  Interface  Control  Document  (ICD)  Complete 

-  Detailed  Design  Compared  to  Code 

-  Software  Testing  Environment  Available 

-  Developer  Tests  &  Results  Shared 


Output  -  Exit  Criteria 


-  Computer  Software  Unit  (CSU),  Computer 
Software  Component  (CSC),  Computer 
Software  Configuration  Item  (CSCI)  or 
Unit/Item  Requirements  Verified 

-  Updated  Allocated  Baseline — Code  and 
Documentation 

-  All  Critical  Requirements  Traced  to  Code 
Level 

-  Approved  Test  Procedures 

-  Computer  Software  Unit  (CSU),  SCS 
&SCSI  or  Unit/Item  Test  Results 

-  Test  Readiness  Review  (TRR)  for  each 
Computer  Software  Configuration  Item 
(CSCI)/Item 

-  Software  Statistics 


Figure  12.  SIV&V  Battle  Rhythm  Code  and  Development  Testing  Phase  (Lewis 

1998) 

6.  Hardware  and  Software  Integration  Phase 

Hardware  and  Software  Integration  (HSI)  verification  testing  has  taken  a 
major  shift  away  from  the  pure  development  environment  to  the  integration 
environment.  Here  the  software  is  typically  developed  on  Functional  Equivalent 
Units  (FEUs)  and  then  moved  to  the  actual  hardware  laboratory  for  integration, 
and  finally  installed  on  the  Field  Unit  Equipped  (FUE)  hardware  for  eventual  use. 
HSI  usually  means  that  a  different  organization  assumes  the  responsibility  for  the 
software  products.  It  also  means  that  the  software  will  run  on  actual  mission 
hardware  or  on  precise  hardware  equivalents  for  the  first  time.  The  following 
tasks  are  typical  of  this  phase  (Lewis  1998): 

•  SIV&V  begins  this  phase  by  reviewing  the  integration  and  test  planning 
documents.  These  typically  describe  the  hardware  environment  in 
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which  the  software  is  to  be  integrated.  Second,  SIV&V  reviews  the 
developer’s  test  documentation  and  reports  to  determine  where  to  best 
focus  SIV&V  resources. 

•  SIV&V  monitors  the  integration  process  and  testing,  and  given  the 
opportunity,  performs  or  shares  independent  integration  level  testing  to 
confirm  the  results  reported  by  the  integrator,  and  explore  new  facets 
and  behaviors  of  the  software,  with  special  emphases  on  critical 
functions  to  ensure  their  adequacy,  correctness,  and  integrity. 

•  In  addition  to  performing  SIV&V  testing,  an  assessment  is  made  of  the 
adequacy  and  completeness  of  developer  and  SIV&V  test  facilities  and 
support  environments. 

•  SIV&V  selects  and  uses  methods,  tools,  and  techniques  for  analysis 
and  evaluation. 

•  An  extremely  important  activity  is  the  verification  and  analysis  of  the 
Software  Product  Specification  (one  per  SI)  including  the  source  code. 
This  is  usually  performed  in  preparation  for  the  Functional 
Configuration  Audit  (FCA). 

•  The  Functional  Configuration  Audit  (FCA)  is  usually  held  when 
hardware  and  software  integration  is  complete  and  the  SI  satisfies  all 
of  the  requirements  levied  upon  them  in  the  Software  Requirement 
Specifications  (SRSs).  However,  this  audit  can,  in  some  cases,  be 
postponed  until  the  next  phase,  following  the  Formal  Qualification 
Tests  (FQTs)  and/or  occasionally  following  a  series  of  flight,  and  live 
tests,  if  required  by  the  contract. 

•  In  any  case,  product  base-lining  cannot  take  place  until  these  audits 
have  been  held  and  all  essential  open  items  are  closed,  otherwise 
deferred,  or  waived. 

•  SIV&V  also  confirms  timing,  sizing,  loading,  and  operating  margins  of 
the  software. 
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•  SIV&V  participates  in  the  component  Test  Readiness  Review  (TRR)  or 
Flight  Readiness  Review  (FRR),  if  held  in  this  phase,  and  reports  the 
results. 

•  Test  beds  are  used  extensively  with  simulations  to  ensure  that  the 
integrated  parts  of  the  system  can  meet  their  stated  performance 
requirements.  Interfaces  are  tested  exhaustively  (e.g.  error  seeding, 
stress  testing,  “what  if”  scenarios,  and  the  like)  to  ensure  that  they  are 
able  to  support  worst  case  loading  and  quality  criteria. 

•  Limited  amounts  of  operational  testing  are  performed  to  ensure  that 
key  algorithms  and  functions  meet  their  deadlines  and  accuracy 
requirements.  These  tests  focus  on  critical  functions,  Measures  Of 
Performance  (MOPs),  and  Measures  Of  Effectiveness  (MOEs)  that  are 
primary  factors  in  the  success  or  failure  of  future  systems  test. 

SIV&V  Battle  Rhythm 
(Hardware  &  Software  Integration 
Phase) 

Input  -  Entrance  Criteria 


n=> 


-  Developer  STPs  and  STDs 

-  Source  &  Executable  Code 

-  Inputs/Outputs  Databases 

-  Results  of  Test  Readiness  Review  (TRR) 

-  Integration  Test  Plans  and  Description 
(ITPs  &  ITDs) 

-  Required  Hardware  Available 

-  Configuration  Management  Records  on 
Software  Anomalies/Deficiencies 


Output  -  Exit  Criteria 

-  Functional,  Perfonnance,  and  Ops 
Requirements  Met  Measure  of  Performance 
(MOPs)  and  Measurement  of  Effectiveness 
(MOEs) 

-  Updates  to  Allocated  Baseline 

-  SPSs  Audited  and  Verified 

-  Human  System  Integration  (HSI)  Complete 

-  All  Critical  Requirements  Evaluated  and 
Verified 

-  STDs  and  ITDs  Updated 

-  Results  of  Reviews  and  Audits 

-  Archived  Test  Results 

-  Flight  Readiness  Review  Held,  if  Required 


Figure  13.  SIV&V  Battle  Rhythm  Hardware  and  Software  Integration  Phase 

(Lewis  1998) 
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7.  Formal  Qualification  Phase 

Validation  is  the  SIV&V  activity  associated  with  the  developer’s  formal 
qualification  testing  and,  in  many  cases  includes  flight  or  live  operational 
certification  of  the  end  item.  This  aspect  of  validation  is  often  carried  forward  into 
the  next  phase.  If  pre-qualified  hardware  already  exists,  formal  validation  will  be 
directed  only  at  the  software;  otherwise,  it  can  involve  both  hardware  and 
software.  However,  SIV&V  typically  focuses  on  the  software  aspects.  The 
following  tasks  are  typical  of  this  phase  (Lewis  1998): 

•  SIV&V  evaluates  qualification  test  suites  for  completeness, 
comprehensiveness,  and  adequacy.  Validation  looks  back  at  system 
and  software  requirements,  determines  adequacy  of  testing,  and 
ensures  that  mission  critical  requirements  are  met. 

•  SIV&V  selects  and  uses  proven  analysis  tools  that  support  the 
performance  assessment.  Then,  SIV&V  authenticates  both  the  data 
and  test  procedures  needed  for  formal  qualification  and,  hence, 
validation. 

•  Validation  is  the  formal  confirmation  or  proof  that  the  system  meets  the 
user’s  expectations,  can  perform  its  mission  to  the  effectiveness 
specified  by  the  MOPs  and  MOEs,  and  that  the  software  is  essentially 
free  of  errors  and  inconsistent  behavior  that  would  affect  its  mission. 
In  a  sense,  validation  is  never  complete  in  that  every  new  use  or 
application  varies  some  of  the  parameters  that  can  affect  the  outcome 
or  operational  integrity  of  the  new  system.  The  objective  is  to  test  and 
validate  beyond  the  nominal  test  cases  (i.e.  off-nominal)  at  a  level  such 
that  the  system  does  not  suddenly  fail  and  can  maintain  performance 
even  when  stressed.  Every  tester  and  validator  struggles  with  where 
to  draw  the  line  to  declare  and  affirm  that  the  system  is  acceptable  for 
use  and  deployment. 

•  SIV&V  analyzes  the  scenarios  and  test  cases  used  for  testing  and 
validation  to  ensure  they  are  sufficient  to  thoroughly  demonstrate  the 


59 


system  capabilities  in  both  nominal  and  off-nominal  (stressing  or  worst 
case)  situations.  Thus,  the  system  is  expected  to  degrade  without 
failing;  therefore,  the  testers  and  validators  have  to  know  where  the 
system  begins  to  degrade  and  what  happens  as  these  limits  are 
exceeded.  These  behaviors  have  a  great  deal  to  do  with  whether  the 
system  is  acceptable  for  use  or  not.  The  software  must  be  designed  to 
accomplish  these  objectives,  and  validation  provides  the  essential 
confirmation. 

•  SIV&V  performs  and  participates  in  Logistical  Reliability  Availability 
Maintainability  (RAM)  testing,  and  effectiveness  evaluation,  as  required 
by  the  customer.  These  are  usually  run  over  extended  periods  of  time. 

•  SIV&V  compares  test  results  to  other  real-world  sources. 

•  SIV&V  uses  Subject  Matter  Experts  (SMEs)  in  the  technology  areas  as 
evaluators. 

•  SIV&V  participates  in  the  Functional  and  Physical  Configuration  Audits 
(FCA  &  PCA),  once  administered,  and  evaluates  and  confirms  that  all 
critical  issues  have  been  resolved  and  closed. 

•  SIV&V  assesses  and  verifies  the  training  of  operators  and  users,  if 
requested  by  the  customer. 

•  All  software  changes  occurring  during  the  period  are  assessed  for  their 
impact  on  source  code  and  base-lined  documentation. 
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SIV&V  Battle  Rhythm 
(Formal  Qualification  Phase) 


Input  -  Entrance  Criteria 


-  Updates  to  System  and  Cl  Architectures 

-  Updates  to  Software  Product  Specification  (SPS) 

-  Fully  Integrated  HWCI  and  CSCIs/SIs  Ready  for  Formal 
Verification  Testing 

-  Updates  to  System  Specification  (SS)  /  Prime  Item 
Development  Specification  (PIDS)  and  Software 
Requirement  Specification  (SRS),  if  required 

-  Product  Baseline 

-  Functional  Configuration  Audit  (FCA)  Results 

-  Flight  Readiness  Review  (FRR)  Results 


Output  -  Exit  Criteria 

-  Verification  of  the  Functional 
Characteristics  of  all  Production  and 
Procurement  Complete 

-  Passage  of  Formal  Qualification  Tests 

-  Verification  of  the  As-Built  Version  of 
CSCIs  Against  SRSs,  SDDs,  IDD  and  DBDD 

-  Approval  of  Software  Product  Specification 
(SPS)s 

-  All  Critical  Functions  Tested 

-  Government  Approval  of  Physical 
Configuration  Audit  (PC  A)  if  held 

-  Establishment  of  CSCI/SI  Product  Baseline 

-  Closure  of  FCA  Open  Items 


Figure  14.  SIV&V  Battle  Rhythm  Formal  Qualification  Phase  (Lewis  1998) 

8.  Operational  Readiness  Phase 

Operational  readiness  validation  is  an  important  SIV&V  phase  of  system 
development  requiring  live  fire,  flight,  or  other  forms  of  operational  status 
validation  or  other  forms  of  pre-operational  checkout  in  a  realistic  “situational” 
environment.  Essentially,  Operational  Readiness  tests  and  demonstrations  take 
on  the  characteristics  of  the  system  they  support.  The  following  tasks  are  typical 
of  this  phase  (Lewis  1998): 

•  To  accomplish  this  form  of  validation,  SIV&V  evaluates  the  limitations 
and  constraints  of  the  system  to  satisfy  all  of  the  operational 
requirements  and  perform  all  of  the  necessary  tests.  SIV&V  may  work 
along  side  the  Operational  Test  and  Evaluation  (OT&E)  personnel  in 
performing  these  tests. 

•  SIV&V  assesses  the  “situational”  test  capabilities  (environment,  test 
ranges,  test  cases,  scenarios,  targets,  behaviors,  and  so  forth)  to 
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ensure  the  proper  mix  of  testing,  the  ability  to  command  and  control 
the  developed  system,  and  general  oversight  of  the  operational  testing. 

•  SIV&V  serves  as  an  operational  expert  for  this  evaluation  together  with 
appropriate  tools  to  support  the  necessary  analysis. 

•  SIV&V  performs  a  self-assessment  based  on  past  and  current  metrics 
that  enable  a  quantitative  measure  of  SIV&V  effectiveness.  SIV&V 
metrics  are  often  compared  to  similar  development  metrics  to 
determine  an  effectiveness  ratio  between  the  subsequent  groups. 

•  All  software  changes  occurring  during  this  phase  undergo  evaluation 
for  their  impact  on  source  code,  design,  and  documentation. 

•  An  operational  test  often  involves  Joint  Services  and  Joint  Level 
components  where  SIV&V  assesses  interoperability  among  the  various 
assets  that  exchange  data,  status,  and  operational  objectives.  SIV&V 
participates  in  these  tests  when  possible  and  ensures  that  operational 
objectives  meet  or  exceed  the  system  expectations. 


SIV&V  Battle  Rhythm 
(Operational  Readiness  Phase) 

Input  -  Entrance  Criteria 


«=> 


-  Functional,  Allocation  and  Product  Baselines 

-  Software  Product  Specification  (SPS)s  and 
Version  Release  Records 

-  Test-Approved  Test  Document 

-  Results  of  Functional  Configuration  Audit 
(FCA)  and  Physical  Configuration  Audit  (PCA) 

-  Integration  Verification  Results 

-  Results  of  Flight  Readiness  Review,  if  required 

-  Operational  Requirements 

-  Operational  Tests  Reports 


Output  -  Exit  Criteria 


-  Results  of  ORR,  if  held 

-  Assessment  of  Various  Operating  and 
Control  Modes,  States  and  Behaviors 

-  Post-Test  Analysis  Results 

-  Effectiveness/Performance  Analysis  Results 

-  Operational  Capabilities  Assessment 

-  Deficiencies/Anomalies  Reported  and 
Closed 


Figure  15.  SIV&V  Battle  Rhythm  Operational  Readiness  Phase  (Lewis  1998) 
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9.  Operations  and  Maintenance  Phase 

In  the  Operations  and  Maintenance  (O&M)  phase,  SIV&V  is  typically  a 
scaled-down  compressed  version  of  the  development  and  integration  phases 
mentioned  previously.  The  key  drivers  for  O&M  SIV&V  are  Engineering  Change 
Proposals  (ECPs)  and  Pre-Planned  Product  Improvement  (P3I)  programs. 
SIV&V  helps  to  evaluate  the  impact  and  magnitude  of  the  proposed  changes  and 
determines  all  the  requirements,  design,  code,  documents,  and  tests  that  are 
affected.  The  following  tasks  are  typical  of  this  phase  (Lewis  1998): 

•  Based  on  the  level  and  types  of  changes,  SIV&V  identifies,  selects, 
and  uses  an  appropriate  mix  of  tools,  techniques,  and  methods  to 
optimize  as  much  as  possible  the  analysis  and  evaluation  of  the 
affected  parts  of  the  software  and/or  system. 

•  SIV&V  is  expected  to  develop  a  tailored  or  scaled  SIV&V  plan,  test 
plan,  and  cost  estimate  as  appropriate  for  the  size  and  schedule  of  the 
proposed  effort. 

•  SIV&V  evaluates  the  changes  for  completeness,  consistency, 
correctness,  impacts  to  other  parts  of  the  system,  and  feasibility. 

•  Typically,  SIV&V  re-verifies  previously  base-lined  documents  and 
code,  examines  the  support  data  and  documentation  such  as  CASE 
and  analytical  tool  outputs  for  completeness,  accuracy,  correctness, 
and  consistency. 

•  SIV&V  performs  testing  in  accordance  with  the  revised  documentation 
(i.e.  Software  Test  Plans  and  Test  Descriptions). 

•  SIV&V  participates  in  all  design  reviews  and  audits  needed  to  assess 
and  evaluate  the  changed  products  at  each  phase. 

•  Once  the  product  has  evolved  to  the  stage  where  formal  re¬ 
qualification  is  needed,  SIV&V  participates  in  the  TRR,  FCA/PCA,  and 
FRR,  as  required  by  the  customer. 

•  All  software  changes  occurring  during  this  period  are  evaluated  for 
their  impacts  on  source  code  and  base-lined  documentation. 
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•  This  phase  usually  covers  scaled-down  versions  of  many  of  the  life 
cycle  phases  and,  therefore,  places  schedule  constraints  on  all  parties: 
customer,  SIV&V,  developers,  and  integrators.  However,  SIV&V’s 
ability  to  meet  any  program  constraints  or  surge  capabilities  is  one  of 
its  major  strengths. 


SIV&V  Battle  Rhythm 
(Operations  and  Maintenance 
Phase) 

Input  -  Entrance  Criteria 


n=> 


Figure  16.  SIV&V  Battle  Rhythm  Operations  and  Maintenance  Phase  (Lewis 

1998) 

D.  SIV&V  SUPPORTING  CASE  TOOLS 

1.  Introduction  (l-Logix.com;  Telelogics;  Spector;  Rational; 
Zambrana) 

Computer-Aided  Software  Engineering  Tools  (CASE)  are  supplemental 
programs  that  assist  in  automating  software-design  and  development  processes. 
For  example  modeling  tools,  compilers,  structure  editors,  and  source-coded 
systems  are  forms  of  CASE  tools.  CASE  tools  relieve  the  programmers  from 
having  to  work  detailed  tasks  related  to  hardware,  thus  allowing  them  to 
concentrate  on  higher-level  software  system  abstractions. 


-  All  Baseline  Documentation  Impacted  by 
Changes 

-  Source  and  Executable  Code 

-  Test  Plan,  Descriptions,  Results  and  Reports 

-  Integration  Test  Results 

-  Configuration  Management  Records 

-  Engineering  Change  Proposals  and  Other 
Change  Requests 

-  Any  Existing  Impact  Analysis 

-  New/Changed  Requirements 


Output  -  Exit  Criteria 

-  Rebaselining  of  All  Affected  Previously 
Baselined  Documentation 

-  Rebaselining  of  Source  Code 

-  Results  of  Development  and  Integration 
Testing 

-  Functional  Configuration  Audit  and  Physical 
Configuration  Audit  Results 

-  Requalification  of  Hardware  Configuration 
Item  and  Computer  Software  Configuration 
Item  /Software  Item 

-  Flight  Readiness  Review  and  Operational 
Readiness  Review  Results,  if  required 

-  Closure  of  Open  Items,  if  required 

-  Complete  Configuration  Management 
Records 
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The  evolutionary  process  of  CASE  tools  has  progressed  to  specific  tools 
of  software  development  that  assist  in  defining  and  validating  particular  aspects 
of  software  system  design.  Current  generations  of  CASE  tools  are  whole 
systems  within  themselves,  consisting  of  multiple  tools  that  assist  software  teams 
with  designing  software  systems  in  a  logical  progressive  pattern. 

There  are  three  types  of  CASE  Tools:  Design,  Hybrids,  and  Build 
environment  tools.  By  definition,  CASE  Design  Tools  assist  diversified  cells  of 
engineers  with  software  system  specification  development.  Design  Tools  further 
assist  engineers  in  code  stubs,  documentation,  and  the  automated  writing  of 
frameworks  that  are  incorporated  into  the  developer’s  program  or  routine.  The 
Unified  Modeling  Language  (UML),  initially  supported  by  Grady  Booch,  Jim 
Rumbaugh,  and  Ivar  Jacobson,  is  another  language  utilized  by  CASE  Design 
Tools.  The  development  of  UMLs  has  revolutionized  the  ability  of  software  to 
produce  system  specifications  that  are  easily  incorporated  into  a  maintainable 
and  productive  code.  The  most  promising  aspect  of  CASE  Design  Tools  is  that 
some  of  them  can  assist  in  the  design  of  almost  unlimited  specifications  from 
document  development  to  embedded  systems  utilized  in  military  Battle 
Management  software. 

CASE  Build  Tools  assist  diversified  cells  of  engineers  in  managing  and 
composing  the  release  of  sophisticated  software  packages;  this  aids  developers 
in  tracking  executables,  objects,  and  combinations  of  sources  encompassed 
within  the  developed  system. 

Hybrid  Tools  are  newly  developed  CASE  Tools  that  combine  existing 
support  tools  with  web  services  to  produce  a  distributed  flexible  system  capable 
of  managing  various  styles  within  software  development  stages.  Hybrid  CASE 
Tools  are  also  capable  of  incorporating  new  software  enhancements  with  a 
minimum  effort  in  labor.  “Sourceforce”  and  “Collab.net”  are  perfect  examples  of 
Hybrid  CASE  Tools,  all  of  which  follow  strict  integration  and  design  processes. 
Further,  they  have  the  capability  to  analyze  the  what,  when,  and  where  of  the 
software  development  process. 
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The  IV&V  change  agent  must  acquire  the  appropriate  software  tools  to 
evaluate  and  address  each  phase  of  the  validation.  Developing  a  large  software 
tool  library  is  useless  unless  all  the  technical  personnel  are  highly  proficient  with 
the  tools  and  their  components. 

2.  Requirements 

Requirements  are  the  compilation  of  specifications  and  activities 
developed  by  the  user  or  customer  in  an  attempt  to  lay  the  guidelines  for  the 
architecture  of  a  developing  system.  In  the  software  community,  a  holistic 
approach  is  used  to  address  requirements.  Once  the  project’s  requirements  are 
developed  and  accepted,  they  are  prioritized  with  respect  to  maximizing  quality, 
maintainability,  and  ease-of-tracking  of  the  software  system.  Therefore, 
elicitation  of  software  requirements  is  a  key  factor  in  the  development  of  new 
software  systems.  Two  important  factors  used  to  define  and  develop  software 
requirements  are  consistency  and  uniformity,  which  are  essential  to  reducing 
system  costs  (Telelogic). 

3.  Design 

The  system  software  design  phase  is  considered  transitional,  in  that  it 
translates  the  software  designers’  concepts,  notions  and  ideas  into  a  semi- 
structured  architecture  and  resourced  entity.  The  design  phase  is  also  iterative 
in  that  it  develops  the  allocation  of  requirements  into  a  specific  design.  Software 
design  is  partitioned  into  two  sub-functions,  the  system  design  and  the  software 
design.  System  design  is  related  to  hardware  development,  and  software  design 
is  related  to  man-in-the-loop  interfaces.  Critical  resources  and  requirements  are 
allocated  during  the  software  design  phase,  and  questions  are  answered 
pertaining  to  the  security  of  data,  maintenance  of  hardware,  software  tools  and 
software  databases.  Examples  of  software  case  tools  utilized  in  the  design 
phase  are  “Rational  Rose,”  “Rhapsody,”  “McCabe,”  and  “Klockwork”  (Spector;  I- 
Logix;  Rational;  Klocwork).] 

4.  Code 

Coding  is  defined  as  translating  the  user’s  requirements  into  a  language 

that  allows  the  computer  to  execute  those  requirements.  Since  “requirements 
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creep”  is  a  consistent  issue,  coding  must  be  able  to  change  with  requirements. 
Thus,  the  logical  assumption  that  requirements  are  clarified  and  finalized  during 
the  coding  process  is  in  fact  rarely  true  for  today’s  complex  systems.  Typically, 
the  design  requirements  are  immature  as  the  coding  process  is  started.  Given 
this  dilemma,  the  programmer  must  try  and  achieve  four  goals  during  the  coding 
process.  First,  quality,  as  the  most  important  goal  in  the  coding  process,  must  be 
emphasized  from  the  beginning  of  the  coding  process  to  assure  that  quality  is 
incorporated  during  the  maintenance  and  test  phases  of  software  development. 
Second,  uniform  and  consistent  guidelines  and  processes  are  required  to  assure 
that  readable  codes  and  production  of  logical  flows  occur.  Third,  coding 
documentation  must  be  in  a  form  that  is  easy  to  understand  and  maintain. 
Programmers  dread  the  maintenance  of  non-standard  codes  as  extreme  efforts 
are  required  to  maintain  these  codes.  Non-standard  codes  typically  result  in 
negligence  of  code  maintenance.  Finally,  the  documentation  is  a  living  document 
that  must  be  iteratively  updated  to  remain  relevant.  As  the  code  matures  the 
documentation  must  become  increasingly  detailed  because  the  documentation 
forms  the  foundation  for  the  operation  and  maintenance  of  the  software  in  a  cost 
and  time  effective  manner  (Shula;  Badeaux). 

5.  Tracking  Database 

A  software  database  is  an  internally  developed  or  Commercial-Off-The- 
Shelf  (COTS)  program  that  tracks  errors  or  bugs  from  initial  to  final  testing  of  the 
software.  The  basic  goals  of  utilizing  a  tracking  database  tool  are  to  find  errors, 
assist  in  correcting  errors,  and  annotate  the  corrections  using  the  report  function 
of  the  database  for  future  reference.  The  bug-tracking  database  is  a  time,  cost, 
and  labor  saving  tool  that  capitalizes  on  the  tracking  and  reporting  functions  of 
the  software.  It  also  enhances  the  programmer’s  abilities  to  discover,  check,  and 
correct  errors  before  the  software  system  enters  into  the  production  phase  of  the 
life  cycle  (Shula). 

6.  Rate  Monotonic  Analysis  (RMA) 

Rate  Monotonic  Analysis  (RMA)  algorithms  were  originally  developed  to 

analyze  the  scheduling  of  periodic  tasks  to  an  associated  periodic  request  rate. 
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The  basic  function  of  RMA  is  to  prioritize  tasks  according  to  the  period  in  which 
they  occur.  In  essence,  tasks  with  shorter-time  periods  are  assigned  a  higher 
priority,  and  tasks  requiring  longer-time  periods  are  assigned  a  lower  priority.  In 
order  to  implement  the  RMA  certain  assumptions  are  made,  for  instance, 
aperiodic  tasks  in  a  system  routine  are  considered  special,  and  will  displace 
periodic  tasks  in  the  execute  routine,  or  there  are  consistent  run-times  for  each 
task,  or  tasks  are  not  dependent  on  the  completion  of  other  tasks.  Task 
deadlines  are  consistently  designed  and  noted  at  the  beginning  of  the  next 
period.  Software  systems  benefit  from  using  RMA  by  retrieving  real-time  task 
conditions  and  analyzing  the  results  statically  to  determine  whether  or  not  task 
deadlines  are  executed,  and  whether  static  scheduling  test  results  fall  within  the 
boundaries  of  the  system’s  rare  monotonic  assumptions  (Forman;  Klien). 

7.  Security  Assessment  (Klocwork;  Ghosh;  Gilliam;  Laliberte) 

Software  applications  that  resident  on  desktops,  mainframes,  and  servers 
are  vulnerable  to  illegal  hacking  and  attacks.  Vulnerabilities  in  computer 
software  arise  from  a  number  of  oversights  in  programming  and  many  times  are 
tracked  back  to  poor  software  development  techniques.  Unsecured  network  links 
and  newly  developed  methods  of  attacks  contribute  to  security  vulnerabilities. 
Security  Assessment  Tools  (SAT)  utilize  current  assessment  methodologies  to 
identify  security  risks  in  resident  and  networked  software.  Vulnerabilities  are 
tested  in  the  client’s  environment,  assessed,  and  prioritized  according  to  the 
estimated  severity  of  effects.  Another  approach  to  assessing  security  is  by 
exploiting  obvious  vulnerabilities  in  break-in  attempts,  and  assessing  whether  or 
not  the  attempts  are  successful.  The  routine  that  allowed  the  break-in  is  patched 
and  the  security  program  searches  other  areas  of  vulnerabilities.  This  method  is 
known  as  penetrate  and  patch. 

Certifying  security  assessments  occur  at  the  component  and  system  level. 
Assessing  and  certifying  at  the  component  level  evaluates  whether  that 
component  operates  as  designed  in  the  operating  environment  without  allowing 
dangerous  penetrations.  At  the  system  level,  software  is  evaluated  at  the  total 
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package  level  for  effectiveness  against  Trojan  horses  or  viruses  and  finally 
assessed  by  SATs  and  individual  evaluations. 


Table  7.  Recommended  SIV&V  Tool  Suite 


TOOL 

Requirements 

Design 

Coding 

Tracking 

Database 

RMA 

Security 

Assessment 

DOORs 

X 

RTM 

X 

Rational 

Rose 

X 

X 

Rhapsody 

X 

McCabe 

X 

X 

Klockwork 

X 

X 

X 

EXCEL 

Spreadsheets 

X 

Flaw  Finder 

X 

Microsoft 

Access 

X 

Timewiz 

X 

Doves 

X 

E.  IMPACTS  OF  ACQUISITION  REFORM 

Acquisition  reform  within  the  Government  has  led  to  an  increase  in  the  use 
of  performance-based  specifications.  This  approach  has  some  advantages  for 
procurement,  but  creates  great  difficulties  for  SIV&V.  The  main  effect  is  that  the 
software  may  be  pushed  to  a  lower  Work  Breakdown  Structure  (WBS)  level  that 
is  not  visible  to  the  Government.  Software  and  software  products  do  not  appear 
in  the  upper  levels  of  the  WBS.  Therefore,  they  may  not  be  delivered  for  review, 
reported  on  schedules  to  the  Government,  or  appear  in  cost  breakouts.  This 
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situation  makes  software  tracking,  oversight,  and  SIV&V  almost  impossible.  The 
response  from  the  contractor  concerning  this  is  that  the  Government  is  welcome 
to  come  to  their  facility  to  review  data  using  their  electronic  systems  such  as 
online  Software  Development  Folders  (SDFs).  Unfortunately,  this  approach  does 
not  work.  Getting  access  to  contractor  facilities  can  be  very  difficult  (badges, 
escorts,  and  so  forth),  and  obtaining  accounts  on  contractor  computer  systems 
can  be  extremely  difficult.  These  issues  are  orders  of  magnitude  more  difficult,  if 
not  impossible,  for  Government  support  contractors  due  to  company  policies, 
non-disclosure  agreements,  proprietary  issues,  and  competition  sensitive 
caveats.  Even  if  access  is  obtained,  locating  the  needed  data  without  assistance 
is  very  difficult  since  the  data  is  often  widely  dispersed  and  poorly  organized. 

The  second  issue  that  is  often  presented  is  that  since  the  contractor  is  a 
Capability  Maturity  Model  (CMM)  Level  3  or  higher  organization,  it  is  assumed 
that  Government  monitoring,  access,  and  even  SIV&V  of  the  software  is  not 
needed.  No  data  or  study  supports  this  position.  In  fact,  the  CMM  promotes 
oversight  and  SIV&V  activities.  A  good  Level  3  and  above  contractor  would 
generate  documents,  have  oversight  and  tracking,  and  provide  status;  however, 
in  the  present  climate  of  budget  cuts,  key  processes  are  often  tailored  out  of  a 
program.  Contractor  management  is  fond  of  saying,  “If  it  is  not  in  the  contract, 
we  don’t  have  to  do  it.”  Even  if  the  CMM  process  is  followed,  Government 
access  may  still  be  limited. 

How  can  this  situation  be  corrected?  The  simple  truth  is  that  if  the 
Government  wants  something,  it  must  be  specifically  stated  in  the  contract.  A 
SIV&V  contract  should  be  awarded  simultaneously  with  the  Prime  contract  and 
should  continue  throughout  the  life  of  the  system.  Document  deliverables  must 
be  identified,  and  provisions  for  providing  SIV&V  access  to  early  release  and 
draft  documents  must  be  included.  Schedule  and  cost  reporting  requirements 
must  include  software  elements.  And  finally,  contractor  support  of  SIV&V 
activities  must  be  indicated.  Adding  SIV&V  into  a  program  late  in  the 
development  cycle  dramatically  reduces  the  effectiveness  and  ROI  of  SIV&V. 
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In  a  similar  vein,  contractor  proprietary  information  creates  great  barriers 
to  SIV&V  activities.  The  contractor  indicates  that  using  an  internally-developed 
proprietary  product  will  save  the  Government  money.  However,  this  is  rarely  the 
case.  Instead,  proprietary  labels  can  be  used  to  limit  Government  visibility  into 
their  activities.  As  a  result,  documents  cannot  be  obtained,  source  code  cannot 
be  inspected,  and  other  contractors  cannot  be  involved.  Often  the  contractor  will 
reuse  one  small  module  within  a  large  new  development,  yet  declare  the  entire 
product  proprietary.  Similarly,  a  document  may  contain  one  paragraph  of 
proprietary  information,  yet  every  page  of  the  document  is  marked  proprietary. 
Non-disclosure  agreements  are  difficult  to  implement  and  do  not  fully  solve  the 
problem.  Challenging  these  cases  is  very  difficult  since  it  immediately  involves 
legal  groups  which  are  costly  in  both  dollars  and  schedule.  Proper  marking  of 
proprietary  information  and  products  to  the  minimum  applicable  object,  and 
minimizing  use  of  proprietary  products  can  help  maximize  the  ROI  on  SIV&V.  In 
addition,  all  proprietary  items  deemed  necessary  must  be  fully  defined  with 
supporting  rationale  during  the  proposal  period.  In  some  areas,  the  Government 
should  request  a  non-proprietary  solution  so  that  Government  ownership  and 
reuse  can  be  obtained.  Scrutiny  must  also  occur  during  the  contract’s  life  to 
make  sure  that  additional  proprietary  items  do  not  appear.  These  measures 
protect  both  the  contractor’s  and  Government’s  rights  to  obtain  a  fair  product 
while  allowing  SIV&V  to  occur  to  the  maximum  extent  possible. 

F.  KEYS  TO  SUCCESSFUL  SIV&V 

To  maximize  ROI,  the  following  key  elements,  that  are  lessons  learned 
from  SED  participation  in  multiple  software  development  and  software  SIV&V 
efforts  during  the  past  decade,  should  be  utilized. 

1)  Government  Program  Managers  (PMs)  should  plan  for  SIV&V  to  be 
included  in  the  initial  phases  of  a  new  development  program  in  order  to  properly 
plan  for  and  execute  a  comprehensive  SIV&V  effort.  A  significant  phase  of 
SIV&V  occurs  during  the  requirements  generation  process,  from  which  SIV&V 
ensures  that  the  necessary  system  requirements/capabilities  are  established  to 
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quantify  the  desired  performance  of  the  new  system.  Many  uninformed  PMs 
mistakenly  assume  that  SIV&V  does  not  need  to  be  included  until  the  software 
implementation  is  complete,  thereby  limiting  SIV&V  to  only  a  software  test  role. 
By  being  involved  early  in  the  software  development  cycle,  SIV&V  is  more 
knowledgeable  about  the  software  products  (documents,  requirements,  and  the 
like)  and  capabilities  of  the  system,  which  will  result  in  a  more  thorough  SIV&V 
effort,  particularly  during  software  testing. 

2)  Software  developed  under  capabilities-based  program  acquisition 
strategies  are  more  difficult  to  verify  and  validate.  The  program  emphasis  on 
capabilities,  rather  than  requirements,  results  in  the  SIV&V  organization 
delineating  between  the  necessary  software,  hardware,  and  personnel 
capabilities  as  part  of  the  overall  system’s  capability.  Most  system  capabilities 
require  a  system-level  test  or  analysis  in  order  to  verify  and  validate,  which 
implies  that  the  SIV&V  organization  should  have  access  to  the  overall  system 
assets  and/or  system  data.  Such  access  is  usually  more  difficult  to  obtain  from  a 
cost  and  schedule  perspective. 

3)  SIV&V  funding  should  be  managed  separately  from  the  organization 
managing  the  software  development  and  other  software  support  functions.  Such 
separation  helps  ensure  that  the  SIV&V  effort  does  not  suffer  financially  from 
software  budget  reductions  and/or  cost  overruns  by  the  Prime  Contractor. 

4)  The  Integrated  Product  Team  (IPT)  concept  can  make  SIV&V  difficult. 
If  SIV&V  is  properly  included  as  part  of  the  IPT,  and  the  IPT  lead  is  the  Prime 
Contractor,  then  the  SIV&V  organization  has  due  allegiance  to  the  Prime 
Contractor,  in  addition  to  the  Government.  Relationships  between  the 
Government,  SIV&V,  and  the  Prime  Contractor  need  to  be  partnered  and  agreed 
upon  prior  to  the  IPT  launch. 

5)  The  SIV&V  organization  must  be  given  necessary  access  to  the 
software-related  materials  (code,  data,  documentation,  and  the  like)  required  to 
perform  SIV&V.  The  Government  must  ensure  that  the  Prime  Contractor  agrees 
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with  such  access.  It  is  best  if  this  agreement  is  established  in  a  written  document 
such  as  the  Prime  Contractor’s  Scope  of  Work  (SOW)  or  Software  Development 
Plan  (SDP).  The  Government  should  be  prepared  to  compensate  the  Prime 
Contractor  for  any  personnel  support  associated  with  responding  to  the  SIV&V’s 
requests  for  information.  This  support  should  not  be  an  additional  cost;  it  should 
be  included  as  part  of  the  Prime  Contractor’s  contract  within  the  SOW. 

6)  The  use  of  the  SEI  CMM  does  not  eliminate  the  need  for  SIV&V.  A 
Software  Development  Organization  (SDO)  can  have  a  high  CMM  rating  but  still 
experience  software  problems.  Some  SDOs  may  choose  to  sidestep  approved 
processes  and  procedures  in  an  attempt  to  cut  costs  or  save  schedule.  The 
SIV&V  organization  is  present  to  ensure  that: 

•  The  software  is  being  built  right. 

•  The  right  software  is  being  built. 

7)  The  SIV&V  contract  should  be  awarded  simultaneously  with  the  Prime 
contract  and  should  continue  through  the  life  of  the  system.  SIV&V  involvement 
should  begin  at  the  beginning  of  the  requirements  stage  so  that  SIV&V  can 
impact  the  maximum  number  of  requirement  errors  during  the  requirements 
phase  when  the  cost  savings  are  greatest.  In  addition,  SIV&V  subject  matter 
experts  are  developed  concurrently  with  developer  experts.  Therefore,  the 
customer  has  two  sources  of  expertise.  Also,  the  developer  can  use  the  SIV&V 
team  as  a  source  of  expertise  during  times  of  personnel  turn-over  and  growth, 
and  when  it  is  more  efficient  to  rely  on  SIV&V  expertise  rather  than  impact  the 
development  schedule.  This  also  positions  the  SIV&V  team  to  continue 
functioning  as  SIV&V  through  Post-Deployment  Software  Support  (PDSS)  or  to 
“be  handed”  the  system  for  maintenance  once  it  no  longer  becomes  cost- 
effective  to  the  developer. 

8)  The  SIV&V  team  should  report  directly  to  the  same  Government  Point 
of  Contact  (POC)  as  the  developer.  This  ensures  that  the  Government  can 
manage  the  relationship  between  the  developer  and  SIV&V  team,  and  make 
certain  that  the  relationship  is  supportive  rather  than  adversarial.  In  addition,  the 
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Government  customer  has  direct  access  to  all  sources  of  information,  which  can 
be  helpful  in  times  of  disparity.  This  also  enforces  the  independence  of  SIV&V. 

9)  SIV&V  must  be  built  into  the  contract  vehicle  of  the  developer.  The 
contract  should  provide  for  SIV&V  to  happen  “in-phase”  or  concurrently.  Draft 
documents  and  build  deliverables  should  be  made  available  to  the  SIV&V  team. 
Also,  SIV&V  should  be  involved  during  the  development  and  testing  cycle.  This 
allows  errors  to  be  found  as  quickly  and  efficiently  as  possible,  and  corrected  in- 
phase  as  often  as  possible,  which  dramatically  decreases  the  cost  to  fix  findings. 
“In-phase”  SIV&V  also  removes  some  of  the  adversarial  relationship  between  the 
developer  and  SIV&V  because  there  is  less  cost  and  schedule  penalty  for  errors 
found,  since  they  are  more  likely  to  be  addressed  in  the  earliest  phase  possible. 
A  close  relationship  of  trust  and  respect  must  be  established  for  the  SIV&V  team 
to  be  involved  at  this  level.  The  SIV&V  team  should  be  viewed  as  a  reliable  and 
cost-effective  resource  by  the  developer. 

10)  The  contract  should  state  that  source  code  will  be  available  to  the 
SIV&V  team.  Source  code  allows  the  SIV&V  team  to  inspect  algorithms  and 
establish  test  cases  that  might  otherwise  go  un-inspected  or  un-tested.  It  also 
allows  the  SIV&V  team  to  test  hard-coded  values  and  make  use  of  commercial 
source  analysis  tools  such  as  McCabe  and  VectorCast. 

11)  SIV&V  should  be  given  a  scope  outside  of  “classical”  SIV&V.  The 
“independence”  in  SIV&V  gives  it  a  natural  tendency  to  excel  in  areas  such  as, 
safety  analysis,  “what  if”  test  scenarios,  and  case  studies.  The  SIV&V  team 
should  use  the  developer’s  tools  to  ensure  “apples-to-apples”  results,  but  also 
should  use  or  develop  test  and  analysis  tools  outside  of  those  used  by  the 
developer.  The  SIV&V  team  should  be  scoped  to  generate  test  cases  to  stress 
the  system  in  unique  and  original  ways.  The  SIV&V  team  should  also  be  scoped 
to  analyze  test  data  using  all  resources  available  to  spot  as  many  anomalies  as 
possible. 
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12)  Often,  software  development  requirements  are  considered  too  late  in 
the  System  Development  process.  System  engineering,  historically,  has  focused 
on  hardware  capability  and  availability.  Hardware  requirements  and  design 
typically  drive  the  System  Development  effort  and  software  often  ends  up  as  an 
afterthought  that  must  operate  with  given  hardware  restrictions.  Software  is  often 
considered  not  to  be  a  critical  component  of  System  Engineering.  Systems  that 
are  considered  to  be  software-intensive  typically  have  limited  staffing  (less  than 
10%  of  the  acquiring  organization)  to  support  the  system  software  acquisition. 
System  Engineering  should  have  both  a  hardware  and  a  software  engineering 
focus.  If  software  engineering  is  not  accounted  for  until  late  in  the  overall  System 
Development  process,  the  software  development  is  reactionary  to  the  hardware 
development,  which  is  often  counterproductive. 

13)  SIV&V  should  include  software  security  assurance  as  part  of  its  scope. 
Due  to  the  constantly  increasing  threat  of  software  intrusions,  the  SIV&V  effort 
should  analyze  the  subject  software  to  identify  and  help  eliminate  any  potential 
software  security  vulnerabilities  or  weaknesses.  DoD  Directive  8570  emphasizes 
the  importance  of  information  security  and  places  the  responsibility  of  software 
security  at  the  respective  Product  Manager  level. 
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IV.  SUMMARY,  RECOMMENDATIONS,  AND  CONCLUSIONS 


A.  SUMMARY 

Over  the  past  60  years,  software  has  become  the  most  critical  component 
of  not  only  every  major  weapons  system  employed  by  the  United  States  Military, 
but  also  serves  as  the  very  foundation  for  the  commercial  infrastructure  (e.g. 
financial,  transportation,  agricultural,  utilities,  medical,  and  the  like)  that  we  as  a 
society  have  come  to  depend  upon  as  we  go  about  our  daily  lives.  Countless 
examples,  some  of  which  have  been  presented  in  this  thesis,  of  software  failures 
and  the  resulting  consequences,  in  lost  time,  resources,  and  regrettably 
sometimes  even  lives,  abound  in  today’s  world.  In  this  respect,  SIV&V  can  be 
viewed  as  “cheap  insurance”  against  the  prospect  of  catastrophic  failures  of 
fielded  software  that  could  possibly  have  tragic  results. 

This  thesis  strives  to  address  this  need  for  better  quality,  highly  reliable 
software,  by  providing  the  reader  with  the  rationale,  guidance,  lessons  learned, 
and  the  tools  necessary  to  solve  these  critical  and  complex  software  dilemmas. 
By  utilizing  the  information  provided  in  this  thesis,  Government  Program 
Managers  and  their  commercial  counterparts  can  significantly  improve  the  odds 
of  fielding  successful  high-quality  reliable  software  products.  It  is  a  process 
where  by  the  user  can  answer  the  key  questions  “Are  we  building  the  right 
thing?”  and  “Are  we  building  the  thing  right?”  Thus,  government  users  can  be 
assured  with  a  high  level  of  certainty  that  their  weapons  systems  will  not  fail  them 
during  a  critical  operational  moment,  and  commercial  users  can  rest  assured  that 
the  very  infrastructure  that  they  have  come  to  depend  on  to  run  our  modern 
society  will  not  collapse. 

B.  CONCLUSIONS 

It  is  essential  that  the  government  agency  or  commercial  entity  and  their 
software  engineering  support  be  administratively  and  technically  competent  to 
effectively  implement  a  quality  software  IV&V  process.  Faults  and  errors  made 
by  software  engineering  support  cause  unnecessary  expenditures  of  time  and 
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money.  The  mishandling  of  these  funds  results  in  an  enmity  towards  the 
organization  that  is  damaging  both  politically  and  institutionally  as  it  affects  the 
ability  of  the  organization  to  complete  the  project.  Beginning  the  project  with  the 
correct  complement  of  skills  for  the  task  precludes  errors,  and  ensures  significant 
system/software  errors  are  surfaced  early  (Makowsky). 

An  in-depth  analysis  has  been  conducted  on  why  it  is  important  to  conduct 
Software  Independent  Verification  and  Validation.  Within  this  thesis,  the  team 
has  documented  examples  of  DoD  and  non-DoD  uses  and  non  uses  of  SIV&V. 
Our  research  solidifies  why  it  is  important  to  use  SI&V  prior  to  deployment  of  a 
new  system  or  product.  The  Software  Engineering  Directorate  (SED),  at  the  U.S. 
Army  Aviation  and  Missile  Command  (AMCOM)  has  extensive  knowledge  in 
reviewing  and  evaluating  software  development,  conducting  design  reviews,  and 
software  integration  and  testing  for  software  builds.  The  example  set  by  SED 
can  serve  as  a  model  for  both  government  and  commercial  entities  that  are 
determined  to  improve  their  software  engineering  processes  through  utilization  of 
SIV&V. 

Finally,  this  thesis  serves  as  a  starting  point  and  tutorial  for 
implementation  of  the  SIV&V  process.  It  provides  a  listing  of  suggested  sources 
of  policy  and  guidance,  suggested  software  computer  CASE  tools,  life  cycle 
phased  descriptions  of  methods  and  criteria  that  can  be  utilized,  and  lessons 
learned  from  other  programs.  It  is  hoped  that  the  reader  will  find  the  information 
within  these  pages  to  be  helpful  and  inspiring  in  their  pursuit  of  software 
excellence. 

C.  RECOMMENDATIONS 

Software  development  and  SIV&V  capabilities  have  increased  significantly 
during  the  past  decade  due  to  the  expanding  role  of  software  in  systems  and 
advances  in  tools  and  processes.  Software  development  productivity  has 
increased  due  to  the  availability  of  integrated  Computer-Aided  Software 
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Engineering  (CASE)  toolsets  that  provide  almost  seamless  transition  between 
the  requirements,  design,  implementation,  and  test  phases  of  the  development 
cycle. 

The  Software  Engineering  Directorate  (SED),  as  a  Government  agency, 
possesses  a  cadre  of  CASE  tools  and  has  an  impressive  advantage  as  a  SIV&V 
resource.  SED  is  a  CMM  Level  IV  organization  for  software  development  and  is 
implementing  the  practices  of  Team  Software  Process  (TSP),  Personal  Software 
Process  (PSP),  and  Capability  Maturity  Model  -  Integrated  (CMMI)  Level  IV. 
Therefore,  SED’s  development  expertise  lends  itself  naturally  to  SIV&V  practices. 
Using  SED  as  the  SIV&V  team  within  the  Army  specifically,  and  DoD  in  general, 
minimizes  conflicts  of  interest  and  proprietary  issues,  and  maximizes 
independence.  In  addition,  SED  is  trusted  by  the  Army  Evaluation  Center  (AEC) 
as  a  resource  to  perform  higher  levels  of  testing,  resulting  in  significant  savings 
to  the  Army.  Many  Army  systems  have  resources  located  in-house,  which  make 
them  available  for  interoperability  testing  using  both  internal  and  external  Army 
and  DoD  network  resources.  This  gives  the  customer  access  to  an  expanded 
tool  set. 

SED  is  also  a  cost-effective  source  of  SIV&V  as  its  customers  can 
leverage  cost  reductions  through  existing  contract  vehicles  which  utilize  onsite 
pricing  structures  and  efficient  operations.  SED  is  the  Life  Cycle  Center  for  the 
ARMY  and  SIV&V,  and  lends  itself  to  the  development  of  experts  that  can 
maintain  a  system  throughout  its  life  cycle. 

Each  military  service  should  consider  establishing  a  “Center  of 

Excellence”  for  SIV&V.  These  centers  would  focus  on  providing  SIV&V  for  each 

of  the  services  programs.  SED  has  a  broad  base  of  SIV&V  experience  and 

capability  that  can  be  made  available  to  other  programs  and  services.  NASA  has 

established  such  a  SIV&V  facility  in  West  Virginia.  This  facility  provides  SIV&V 

support  to  all  NASA  centers  utilizing  both  Government  and  contractor  personnel. 

This  concept  of  SIV&V  “Centers  of  Excellence”  can  be  extended  to  the 

commercial  software  industry  as  well.  Similar  to  the  engineering  and  standards 
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societies  that  have  founded  to  support  various  industries  like  IEEE,  ANSI  and  the 
like,  these  SIV&V  “Centers  of  Excellence”  could  be  organized  by  industry  and/or 
criticality  of  function  or  service  that  the  software  is  providing.  Examples  of 
“Centers  of  Excellence”  here  would  be  medical  software,  aviation  software, 
nuclear  power  software,  or  financial  software. 

A  follow-on  concept  definition  study  should  be  performed  to  make 
recommendations  regarding  the  organization,  management,  and  execution  of  the 
proposed  SIV&V  “Centers  of  Excellence.” 

D.  ANSWERS  TO  RESEARCH  QUESTIONS 

1.  Primary  Research  Question 

What  are  the  benefits  of  and  rationale  for  PMOs  and  others  for  using 
SIV&V? 

See  Chapter  I  Sections  B  and  G,  Chapter  II  Sections  A,  Cl,  and  C3,  and 
Chapter  III  Section  B. 

2.  Secondary  Research  Questions 

What  software  CASE  tools  are  available  for  software  V&V? 

See  Chapter  III,  Section  D,  for  a  listing  of  SIV&V  CASE  Tools. 

What  key  things  should  be  done  or  considered  when  conducting  SIV&V? 
When  conducting  SIV&V  it  is  important  to  do  or  consider  the  following  key 

things: 

•  Document  the  organizational  structures  and  processes  that  support 
centralized  coordination  of  SIV&V  activities  and  deliverables  to  achieve 
maximum  efficiency. 

•  Standardize  SIV&V  policies,  processes,  and  products  across  the 
program. 

•  Identify  technical  issues  and  trends  in  a  timely  manner  for 
management  focus. 

•  Integrate  and  enhance  existing  SIV&V  activities  within  the  SIV&V 
community. 
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•  Provide  leadership  insight  into  SIV&V  plans,  milestones,  and  progress. 

•  Consider  how  SIV&V  will  reduce  risk  of  program,  element  or 
component  failure  due  to  operational  baseline  or  simulation  software 
defect(s). 

•  Develop  and  test  software  to  ensure  high  confidence  in  the  system  and 
component  capabilities. 

•  Ensure  software  is  mature  and  dependable. 

•  Ensure  SIV&V  is  executed  via  the  coordinated  efforts  of  all  program 
directorates  and  component  project  offices. 

•  Ensure  SIV&V  activities  examine  the  Prime  contractor’s  software 
development  plans,  processes,  and  products  throughout  the  software 
development  lifecycle. 

•  Ensure  early  identification  of  products  for  delivery,  otherwise,  your 
support  may  not  provide  any  added  benefit. 

•  Identify  coordination  requirements  and  information  flow  for  access  to 
software,  documentation,  and  data  items. 

•  Ensure  early  identification  of  tracking  mechanisms  and  tools  as  well  as 
establishing  configuration  status  accounting  processes. 

What  are  the  SIV&V  process  steps? 

See  Chapter  III,  Applying  SIV&V  In  The  Cycle  Phase. 

Another  excellent  source  is  the  SEES  process.  The  U.  S.  Army  Aviation 
and  Missile  Command's  Missile  Research,  Development,  and  Engineering 
Center  (AMRDEC),  Software  Engineering  Directorate  (SED),  has  developed  the 
Software  Engineering  Evaluation  System  (SEES).  The  SEES  defines  the 
Independent  Verification  and  Validation  (IV&V)  tasks,  including  procedures  for 
crucial  unique  software  development  issues  that  may  be  performed  by  the  SED 
in  support  of  a  Program  Management  Office  (PMO)  request  to  evaluate  software 
intensive  systems.  The  overall  SEES  approach,  based  upon  DOD-STD-2167A 
[1],  is  depicted  on  the  Integrated  System  Diagram  (ISD)  provided  in  APPENDIX 
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A.  The  SEES  utilizes  analysis  methods  and  practices  compatible  with  the 
developer's  effort.  The  results  of  the  SEES  methods  and  practices  document  the 
software  engineering  accuracy,  as  well  as  the  deficiencies  of  the  developmental 
products. 

How  has  acquisition  reform  affected  SIV&V? 

See  Chapter  III,  Section  E. 

What  lessons  have  been  learned  from  past  programs  that  have  utilized 
SIV&V? 

In  most  cases,  the  SIV&V  agent  tries  to  maintain  their  independence  while 
staying  intimately  involved  in  the  day-to-day  software  activities  performed  by  the 
prime  developer.  If  this  were  not  the  case,  the  government  assessments  would 
be  limited  to  the  developer’s  test  program  only.  This  concept  is  a  big  concern 
because  the  software  will  be  executed  in  ways  that  the  rule-writers  do  not  fully 
cover.  Accordingly,  the  developer  decides  what,  when,  where,  and  how  the 
software  will  be  tested.  The  developer  also  decides  what  and  when  test  data  will 
be  available  for  government  review.  Simply  put  the  SIV&V  agent  provides 
oversight  for  the  developer’s  test  plans,  procedures,  testing,  and  data  analysis. 

Another  short  fall  or  lesson  learned  is  that  if  SIV&V  is  not  performed  then 
the  Government  will  have  no  software  risk  mitigation  capabilities  for  defect 
identification  and  prevention.  This  means  the  government  cannot  independently 
address  high-risk  areas  (interfaces,  critical  algorithms,  and  the  like),  perform  any 
accelerated  software  checkout  activities  or  conduct  additional  regression  testing, 
or  respond  to  other  element  IV&V  test  concerns  or  findings. 

Yet  another  lesson  learned  is  that  there  are  large  risks  associated  with 
utilizing  a  reduced  SIV&V  or  tailored  team.  The  consequences  of  this  are  as 
follows: 

•  Software  requirements  become  program  risk  items,  as  there  is  no 
Government  insight  into  missing  or  incomplete  requirements, 
traceability  or  flow  down,  or  requirements  certification. 
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•  Software  design  processes  become  questionable,  as  there  is  no 
Government  contribution  to  the  software  design  documentation,  and  no 
subject  matter  experts  for  software  design  and  implementation 

•  Software  Testing  and  integration  become  high-risk  items,  as  there  is 
no  “on-the-ground”  insight  or  test  witnessing  (regarding  status)  of 
FQTs,  internal  integration  events,  or  field  software  check-outs.  This  is 
why  SIV&V  is  declared  “cheap  insurance”  for  the  project  offices,  as 
they  are  solely  dependent  upon  SIV&V  for  solutions  to  software 
problems. 

E.  RECOMMENDATIONS  FOR  FURTHER  STUDY 

Good  software  should  be  cost  effective,  reliable,  maintainable,  defect  free 
and  above  all  usable.  Presently  software  possess  very  few  of  these  attributes. 
Software  is  simply  dreadful  today  and  is  becoming  shoddier  all  the  time.  Industry 
and  the  government  need  to  partner  to  find  better  ways  to  educate  people  both 
within  government  and  industry  on  the  necessity  of  SIV&V  and  its  use. 
Accordingly,  the  software  community  should  continue  to  develop  and  evolve  the 
SIV&V  process  and  tools  to  keep  pace  with  software  evolution  and  its  increasing 
complexities.  One  such  area  that  should  be  considered  for  further  study  is  called 
Independent  Integrated  Verification  and  Validation  (l2V2)  (Dalrymple).  I2V2  is  an 
evolution  of  IV&V  that  suggests  that  SIV&V  agents  should  become  an  integral 
part  of  the  acquisition  process  by  becoming  involved  at  the  very  beginning  of  the 
development  process.  How  this  would  be  accomplished  and  how  the  SIV&V 
agents  would  retain  their  objectivity  and  independence  are  key  questions  that 
need  to  be  answered.  Failure  of  the  software  industry  to  rise  to  this  challenge  in 
conjunction  with  the  increasingly  litigious  nature  of  our  society,  may  very  well 
lead  to  expensive  litigation  as  lawyers  and  special  interest  groups  discover  yet 
another  lucrative  commercial  enterprise  ripe  for  exploitation. 
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SEES  Generated  Intmmation  DoD-STD-2!67A  DIOs 


APPENDIX:  INTEGRATED  SYSTEM  DIAGRAM  (ISD) 


Software  EnApeering  Directorate 

Software  Engiraering  Evaluation  System 

(SEES) 
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